Sep 10 20:36:02.090837 ip-10-0-129-103 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:36:02.503469 ip-10-0-129-103 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:36:02.503469 ip-10-0-129-103 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:36:02.503469 ip-10-0-129-103 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:36:02.503469 ip-10-0-129-103 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:36:02.503469 ip-10-0-129-103 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:36:02.505341 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.505268    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:36:02.509422 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509407    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:02.509422 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509422    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509426    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509429    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509434    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509438    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509441    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509444    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509447    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509450    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509453    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509456    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509459    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509461    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509465    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509467    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509470    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509472    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509482    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509485    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:02.509491 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509488    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509491    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509493    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509496    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509504    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509507    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509510    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509513    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509515    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509518    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509522    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509525    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509528    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509531    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509533    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509536    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509539    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509543    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509547    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509550    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:02.509966 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509553    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509568    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509572    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509574    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509577    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509579    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509582    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509585    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509587    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509589    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509592    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509594    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509597    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509600    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509603    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509606    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509609    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509612    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509615    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:02.510496 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509618    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509620    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509623    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509625    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509628    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509630    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509633    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509635    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509638    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509640    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509643    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509645    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509648    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509651    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509653    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509656    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509659    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509661    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509664    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509667    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:02.510983 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509669    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509672    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509674    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509677    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509679    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509681    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509684    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509687    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509690    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509692    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509695    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509698    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509700    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509703    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509706    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509708    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509711    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509713    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509716    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.509718    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:02.511464 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510377    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510382    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510385    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510388    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510391    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510393    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510396    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510399    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510403    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510410    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510413    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510416    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510418    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510421    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510424    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510427    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510430    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510432    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510436    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510440    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:02.511968 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510442    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510445    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510448    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510451    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510453    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510456    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510459    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510461    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510464    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510466    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510469    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510471    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510474    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510476    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510479    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510482    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510484    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510487    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510489    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510492    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:02.512550 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510494    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510497    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510500    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510503    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510505    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510508    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510510    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510513    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510515    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510518    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510521    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510525    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510527    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510530    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510533    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510535    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510538    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510541    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510543    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:02.513102 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510546    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510548    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510551    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510554    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510570    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510573    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510575    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510578    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510580    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510583    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510586    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510588    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510591    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510593    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510596    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510598    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510602    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510604    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510607    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510609    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:02.513617 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510612    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510614    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510617    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510620    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510624    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510628    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510632    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510636    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510639    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510642    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510645    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510647    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510650    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510653    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510656    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510659    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510662    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510665    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510667    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:02.514109 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.510669    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511640    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511648    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511651    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511654    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511657    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511660    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511663    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511665    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511668    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511670    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511673    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511676    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511678    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511681    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511683    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511686    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511689    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511691    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511694    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:02.514609 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511696    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511699    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511701    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511705    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511709    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511712    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511715    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511718    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511720    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511723    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511725    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511728    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511731    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511734    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511736    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511738    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511741    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511743    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511746    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:02.515103 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511749    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511751    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511754    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511757    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511760    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511762    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511765    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511768    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511771    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511773    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511776    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511779    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511781    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511784    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511787    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511789    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511792    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511794    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511797    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:02.515578 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511799    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511801    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511804    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511806    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511809    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511812    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511815    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511817    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511819    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511822    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511824    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511827    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511831    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511833    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511835    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511838    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511841    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511844    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511846    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511849    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:02.516060 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511851    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511854    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511856    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511859    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511862    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511864    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511867    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511869    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511872    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511874    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511877    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511879    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511882    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511884    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511887    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511889    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511892    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511896    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511899    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511902    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:02.516542 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511904    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.511907    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.511975    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512016    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512024    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512028    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512032    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512035    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512039    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512043    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512048    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512051    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512054    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512057    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512060    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512063    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512066    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512069    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512072    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512075    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512079    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512082    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512085    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512088    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:36:02.517052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512091    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512095    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512098    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512101    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512104    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512107    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512110    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512112    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512116    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512119    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512122    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512125    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512128    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512131    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512134    2565 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512137    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512141    2565 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512144    2565 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512148    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512153    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512156    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512160    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512162    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512165    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512169    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:36:02.517653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512171    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512174    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512177    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512180    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512183    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512186    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512201    2565 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512205    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512209    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512212    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512215    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512218    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512221    2565 flags.go:64] FLAG: --help="false"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512224    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512227    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512230    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512233    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512237    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512240    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512243    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512246    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512249    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512252    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:36:02.518265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512254    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512257    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512260    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512266    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512270    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512273    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512275    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512278    2565 flags.go:64] FLAG: --lock-file=""
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512281    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512284    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512287    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512292    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512295    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512298    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512301    2565 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512304    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512307    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512310    2565 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512314    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512318    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512321    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512325    2565 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512328    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512331    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512333    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:36:02.518866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512336    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512339    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512342    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512345    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512352    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512355    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512358    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512361    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512364    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512369    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512371    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512376    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512379    2565 flags.go:64] FLAG: --port="10250"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512382    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512385    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09d8f5a52af1993f9"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512388    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512391    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512394    2565 flags.go:64] FLAG: --register-node="true"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512397    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512402    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512405    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512407    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512410    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512414    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512417    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:36:02.519477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512420    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512422    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512425    2565 flags.go:64] FLAG: --runonce="false"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512428    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512431    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512434    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512437    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512440    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512443    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512446    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512449    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512452    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512455    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512458    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512460    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512463    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512466    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512469    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512476    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512478    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512482    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512485    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512488    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512491    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512494    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:36:02.520101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512497    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512501    2565 flags.go:64] FLAG: --v="2"
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512505    2565 flags.go:64] FLAG: --version="false"
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512508    2565 flags.go:64] FLAG: --vmodule=""
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512512    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.512515    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512639    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512644    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512647    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512650    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512654    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512656    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512659    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512663    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512666    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512670    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512673    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512675    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512678    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512680    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512683    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:02.520735 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512686    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512689    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512691    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512694    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512698    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512701    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512703    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512706    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512709    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512712    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512714    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512717    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512720    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512723    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512726    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512728    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512731    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512733    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512736    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:02.521246 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512738    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512741    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512743    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512746    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512749    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512751    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512754    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512756    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512759    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512761    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512764    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512766    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512769    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512772    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512774    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512777    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512779    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512783    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512785    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512788    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:02.521798 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512790    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512793    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512796    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512798    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512801    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512805    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512807    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512810    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512813    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512815    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512818    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512821    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512823    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512826    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512829    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512831    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512834    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512836    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512839    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512843    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:02.522320 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512846    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512850    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512852    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512856    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512859    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512862    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512865    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512868    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512871    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512875    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512878    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512881    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512883    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512886    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512889    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512891    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512894    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512897    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512900    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:02.522906 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512902    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512905    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512908    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512910    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512913    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.512915    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.514168    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.521632    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.521646    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.522487    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.522494    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522548    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522553    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522570    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522575    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:02.523383 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522578    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522581    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522584    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522586    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522589    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522591    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522594    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522597    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522600    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522603    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522606    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522608    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522612    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522614    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522617    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522620    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522622    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522625    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522628    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522630    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:02.523763 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522633    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522635    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522638    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522641    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522645    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522649    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522653    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522656    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522658    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522661    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522664    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522666    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522669    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522671    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522675    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522677    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522680    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522683    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522686    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:02.524268 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522688    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522691    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522694    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522697    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522699    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522702    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522705    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522708    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522711    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522713    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522716    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522718    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522721    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522724    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522727    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522729    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522732    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522735    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522738    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522741    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:02.524738 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522743    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522746    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522749    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522751    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522754    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522758    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522761    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522764    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522767    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522770    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522773    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522776    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522779    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522782    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522784    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522787    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522790    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522793    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522796    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:02.525216 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522799    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522802    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522805    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522808    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522810    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522813    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522817    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522820    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522823    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522825    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522829    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522831    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522834    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522836    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522839    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522841    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:02.525770 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522844    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.522849    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522975    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522981    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522984    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522987    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522990    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522993    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522995    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.522998    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523000    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523003    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523006    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523008    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523010    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:02.526157 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523013    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523016    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523019    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523021    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523025    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523029    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523032    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523034    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523037    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523040    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523044    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523047    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523050    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523053    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523056    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523059    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523061    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523064    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523067    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:02.526526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523069    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523071    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523075    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523077    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523080    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523083    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523085    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523088    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523091    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523094    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523096    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523099    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523101    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523104    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523106    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523109    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523112    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523114    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523117    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523119    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:02.527059 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523122    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523124    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523127    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523129    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523132    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523134    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523137    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523139    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523142    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523144    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523147    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523149    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523152    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523154    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523157    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523159    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523162    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523165    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523167    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523170    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:02.527520 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523172    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523175    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523177    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523180    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523182    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523184    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523187    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523190    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523192    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523197    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523200    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523202    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523205    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523207    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523210    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523212    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523215    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523217    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523220    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523222    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:02.528025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523225    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523227    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523230    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523232    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523235    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523237    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:02.523241    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.523246    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.523333    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.525665    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.526818    2565 server.go:1019] "Starting client certificate rotation"
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.526912    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:36:02.528518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.526979    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:36:02.529720 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.529707    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:36:02.530830 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.530813    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:36:02.540900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.540879    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:36:02.544163 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.544148    2565 log.go:25] "Validated CRI v1 image API"
Sep 10 20:36:02.545056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.545038    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:36:02.550226 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.550208    2565 fs.go:135] Filesystem UUIDs: map[576bf228-7493-49d3-9a94-1efea43ed386:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ca8ed389-3d53-4ae7-9f3e-e4fe61c5eaa5:/dev/nvme0n1p3]
Sep 10 20:36:02.550286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.550225    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:36:02.555304 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.555177    2565 manager.go:216] Machine: {Timestamp:2026-09-10 20:36:02.553730502 +0000 UTC m=+0.362893643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3077836 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dea59f2bcfb610233087d7392fa1d SystemUUID:ec2dea59-f2bc-fb61-0233-087d7392fa1d BootID:c63d099c-9a96-4580-9488-ee4782142dd5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:99:2a:6b:a8:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:99:2a:6b:a8:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:36:cc:ae:94:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:36:02.555304 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.555295    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:36:02.555444 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.555364    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:36:02.558805 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.558781    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:36:02.558930 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.558808    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:36:02.559663 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.559654    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:36:02.559701 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.559667    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:36:02.559701 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.559674    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:36:02.560331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.560313    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:36:02.560477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.560467    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:36:02.561713 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.561703    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:36:02.567233 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567221    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:36:02.567340 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567331    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:36:02.567382 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567359    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:36:02.567382 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567369    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:36:02.567446 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567383    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:36:02.567446 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567392    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:36:02.567446 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567400    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:36:02.567713 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567699    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:36:02.567748 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.567720    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:36:02.570019 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.570002    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:36:02.572882 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.572868    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:36:02.572943 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.572898    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:36:02.574873 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574861    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:36:02.574873 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574877    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574892    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574898    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574906    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574912    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574926    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574931    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574936    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574941    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574953    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:36:02.574981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.574959    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:36:02.575864 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.575852    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:36:02.575902 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.575867    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:36:02.578603 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.578583    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:36:02.579155 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.579137    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:36:02.579229 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.579210    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:36:02.579419 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.579406    2565 server.go:1268] "Started kubelet"
Sep 10 20:36:02.579509 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.579476    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:36:02.579543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.579487    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:36:02.579595 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.579549    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:36:02.580243 ip-10-0-129-103 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:36:02.584872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.584850    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:36:02.586414 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.586312    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:36:02.586770 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.586754    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:36:02.592046 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.592027    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:36:02.592612 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.592596    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:36:02.592699 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.592682    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:02.593300 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593282    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:36:02.593300 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593302    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:36:02.593413 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593281    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:36:02.593458 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593428    2565 factory.go:55] Registering systemd factory
Sep 10 20:36:02.593458 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593441    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:36:02.593582 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593467    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:36:02.593582 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593444    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:36:02.593820 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593804    2565 factory.go:153] Registering CRI-O factory
Sep 10 20:36:02.593820 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593821    2565 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:36:02.593955 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593870    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:36:02.593955 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593886    2565 factory.go:103] Registering Raw factory
Sep 10 20:36:02.593955 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.593899    2565 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:36:02.594274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.594260    2565 manager.go:349] Starting recovery of all containers
Sep 10 20:36:02.597868 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.597838    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 20:36:02.598025 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.597996    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 20:36:02.598906 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.597734    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-103.ec2.internal.18d4100449f119b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-103.ec2.internal,UID:ip-10-0-129-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-103.ec2.internal,},FirstTimestamp:2026-09-10 20:36:02.579380657 +0000 UTC m=+0.388543798,LastTimestamp:2026-09-10 20:36:02.579380657 +0000 UTC m=+0.388543798,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-103.ec2.internal,}"
Sep 10 20:36:02.604943 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.604809    2565 manager.go:354] Recovery completed
Sep 10 20:36:02.606200 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.606179    2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 20:36:02.609359 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.609345    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:02.612259 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.612239    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:02.612338 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.612277    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:02.612338 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.612292    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:02.612820 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.612803    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:36:02.612820 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.612821    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:36:02.612917 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.612810    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:02.612917 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.612838    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:36:02.613619 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.613580    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:02.613619 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.613608    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:02.613749 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.613636    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:02.614214 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.614152    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-103.ec2.internal.18d410044be6c68d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-103.ec2.internal,UID:ip-10-0-129-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-103.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-103.ec2.internal,},FirstTimestamp:2026-09-10 20:36:02.612258445 +0000 UTC m=+0.421421589,LastTimestamp:2026-09-10 20:36:02.612258445 +0000 UTC m=+0.421421589,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-103.ec2.internal,}"
Sep 10 20:36:02.615080 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.615065    2565 policy_none.go:49] "None policy: Start"
Sep 10 20:36:02.615080 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.615083    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:36:02.615219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.615095    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:36:02.617057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.617044    2565 policy_none.go:47] "Start"
Sep 10 20:36:02.628119 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.628064    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-103.ec2.internal.18d410044be72d68  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-103.ec2.internal,UID:ip-10-0-129-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-103.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-103.ec2.internal,},FirstTimestamp:2026-09-10 20:36:02.612284776 +0000 UTC m=+0.421447921,LastTimestamp:2026-09-10 20:36:02.612284776 +0000 UTC m=+0.421447921,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-103.ec2.internal,}"
Sep 10 20:36:02.628359 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.628343    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bwzpc"
Sep 10 20:36:02.635648 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.635630    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bwzpc"
Sep 10 20:36:02.642451 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.642362    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-103.ec2.internal.18d410044be761d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-103.ec2.internal,UID:ip-10-0-129-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-103.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-103.ec2.internal,},FirstTimestamp:2026-09-10 20:36:02.612298194 +0000 UTC m=+0.421461339,LastTimestamp:2026-09-10 20:36:02.612298194 +0000 UTC m=+0.421461339,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-103.ec2.internal,}"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.656109    2565 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.656139    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.656151    2565 server.go:87] "Starting device plugin registration server"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.656410    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.656419    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.656527    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.656634    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.656647    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.657067    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:36:02.666810 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.657110    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:02.684490 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.684465    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:36:02.685650 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.685629    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:36:02.685722 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.685678    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:36:02.685722 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.685699    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:36:02.685813 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.685766    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:36:02.687579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.687551    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:02.756887 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.756824    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:02.757591 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.757549    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:02.757670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.757599    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:02.757670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.757609    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:02.757670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.757628    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.765813 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.765792    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.765813 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.765813    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-103.ec2.internal\": node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:02.785894 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.785874    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"]
Sep 10 20:36:02.785965 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.785929    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:02.786636 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.786622    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:02.786688 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.786645    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:02.786688 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.786655    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:02.788987 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.788976    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:02.789126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.789165 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789139    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:02.789664 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789648    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:02.789725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789664    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:02.789725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789675    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:02.789725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789684    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:02.789725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789687    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:02.789725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.789697    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:02.791922 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.791908    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.791973 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.791932    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:02.792543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.792527    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:02.792614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.792553    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:02.792614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.792579    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:02.815073 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.815048    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-103.ec2.internal\" not found" node="ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.818337 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.818324    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-103.ec2.internal\" not found" node="ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.894638 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.894618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ab19bc1a57527adb982141bacf53b44-config\") pod \"kube-apiserver-proxy-ip-10-0-129-103.ec2.internal\" (UID: \"2ab19bc1a57527adb982141bacf53b44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.894723 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.894645    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/854af0dc85e6ffa23f95a26bb14f0ae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"854af0dc85e6ffa23f95a26bb14f0ae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.894723 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.894663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/854af0dc85e6ffa23f95a26bb14f0ae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"854af0dc85e6ffa23f95a26bb14f0ae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.900633 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:02.900617    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:02.995403 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.995378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/854af0dc85e6ffa23f95a26bb14f0ae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"854af0dc85e6ffa23f95a26bb14f0ae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.995457 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.995450    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/854af0dc85e6ffa23f95a26bb14f0ae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"854af0dc85e6ffa23f95a26bb14f0ae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.995497 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.995407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/854af0dc85e6ffa23f95a26bb14f0ae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"854af0dc85e6ffa23f95a26bb14f0ae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.995497 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.995482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/854af0dc85e6ffa23f95a26bb14f0ae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"854af0dc85e6ffa23f95a26bb14f0ae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.995574 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.995493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ab19bc1a57527adb982141bacf53b44-config\") pod \"kube-apiserver-proxy-ip-10-0-129-103.ec2.internal\" (UID: \"2ab19bc1a57527adb982141bacf53b44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:02.995574 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:02.995532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ab19bc1a57527adb982141bacf53b44-config\") pod \"kube-apiserver-proxy-ip-10-0-129-103.ec2.internal\" (UID: \"2ab19bc1a57527adb982141bacf53b44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:03.001051 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:03.001038    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:03.101767 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:03.101708    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:03.117778 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:03.117762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:03.121277 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:03.121261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:03.202525 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:03.202504    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:03.302984 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:03.302962    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:05.301657 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.301641    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Sep 10 20:36:05.338681 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.338662    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:05.391352 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.391337    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:05.405267 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.405245    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:36:05.406695 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.406670    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Sep 10 20:36:05.423218 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.423198    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:36:05.424264 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.424251    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:36:05.424367 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.424352    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:36:05.424403 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.424385    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:36:05.444381 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.444365    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:05.465995 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.465979    2565 apiserver.go:52] "Watching apiserver"
Sep 10 20:36:05.477193 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.477176    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:36:05.477565 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.477548    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal","openshift-cluster-node-tuning-operator/tuned-twhdb","openshift-dns/node-resolver-vzjhd","openshift-image-registry/node-ca-glc9l","openshift-multus/multus-additional-cni-plugins-6nvt4","openshift-multus/network-metrics-daemon-kjnmq","openshift-network-operator/iptables-alerter-6s6vq","kube-system/global-pull-secret-syncer-twjd6","kube-system/konnectivity-agent-n7fzz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal","openshift-multus/multus-9f7gx","openshift-network-diagnostics/network-check-target-ppwj5","openshift-ovn-kubernetes/ovnkube-node-jwj4x"]
Sep 10 20:36:05.480124 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.480109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.482334 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.482317    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:05.482409 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.482374    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:05.483480 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.483466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:36:05.484304 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.484266    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:36:05.484416 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.484366    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4qltg\""
Sep 10 20:36:05.484514 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.484499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.484727 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.484712    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:36:05.486710 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.486679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.488917 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.488902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.490670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.490656    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:36:05.491736 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.491712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:05.491809 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.491789    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:05.493587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493056    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:36:05.493587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493091    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:36:05.493587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493101    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:36:05.493587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493365    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:36:05.493587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493531    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:36:05.493881 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493635    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:36:05.493881 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493727    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:36:05.493881 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.493799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:36:05.495133 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.494189    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:36:05.495133 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.494224    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn8rf\""
Sep 10 20:36:05.495133 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.494406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.495133 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.494764    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qj45q\""
Sep 10 20:36:05.495501 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.495448    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:36:05.496081 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.495611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q78tm\""
Sep 10 20:36:05.497283 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.497627 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqkdb\" (UniqueName: \"kubernetes.io/projected/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-kube-api-access-dqkdb\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.497627 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwtc\" (UniqueName: \"kubernetes.io/projected/9171816d-e76b-42c2-bb56-e341aaed3db4-kube-api-access-tdwtc\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:05.497627 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-cnibin\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.497627 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b390050-29b1-4801-891a-7dcd21d92725-iptables-alerter-script\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.497627 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-hosts-file\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0d0e19d-c65f-4d91-9151-8340d44d2502-host\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-system-cni-dir\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7738d02-740f-4e53-9c3d-bcce706af7c7-konnectivity-ca\") pod \"konnectivity-agent-n7fzz\" (UID: \"b7738d02-740f-4e53-9c3d-bcce706af7c7\") " pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-tmp-dir\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-cni-binary-copy\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66mq\" (UniqueName: \"kubernetes.io/projected/9b390050-29b1-4801-891a-7dcd21d92725-kube-api-access-z66mq\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0d0e19d-c65f-4d91-9151-8340d44d2502-serviceca\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhvx\" (UniqueName: \"kubernetes.io/projected/c0d0e19d-c65f-4d91-9151-8340d44d2502-kube-api-access-qjhvx\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:05.497931 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.498561 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.497969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhpt\" (UniqueName: \"kubernetes.io/projected/a2bde71e-aa70-43ef-8b80-61755a1e7085-kube-api-access-dxhpt\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.498561 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.498016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7738d02-740f-4e53-9c3d-bcce706af7c7-agent-certs\") pod \"konnectivity-agent-n7fzz\" (UID: \"b7738d02-740f-4e53-9c3d-bcce706af7c7\") " pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.498561 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.498044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b390050-29b1-4801-891a-7dcd21d92725-host-slash\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.499509 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.499492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.500258 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.500243    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:36:05.501913 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.501895    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.501992 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.501973    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4fw7w\""
Sep 10 20:36:05.502348 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.502335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:36:05.502516 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.502504    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:36:05.504030 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.504015    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:36:05.504331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.504312    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:36:05.504394 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.504359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:36:05.504433 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.504362    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:36:05.504895 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.504881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.504953 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.504938    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:05.505359 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.505349    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:36:05.505462 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.505446    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:36:05.505509 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.505499    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:36:05.505641 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.505629    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvbqz\""
Sep 10 20:36:05.505679 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.505664    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:36:05.505679 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.505672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:36:05.505976 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.505962    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-28xcw\""
Sep 10 20:36:05.506037 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.506021    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:36:05.507146 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.507131    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.507545 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.507530    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kt447\""
Sep 10 20:36:05.513841 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.513818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:36:05.514175 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.514159    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gdq2c\""
Sep 10 20:36:05.518295 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.518278    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:36:05.523006 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.522630    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:05.540639 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.540618    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:31:04 +0000 UTC" deadline="2028-06-19 02:49:47.098308822 +0000 UTC"
Sep 10 20:36:05.540639 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.540638    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15534h13m41.557672676s"
Sep 10 20:36:05.543767 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.543747    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hktv"
Sep 10 20:36:05.559527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.559485    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hktv"
Sep 10 20:36:05.592124 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.592105    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:36:05.598219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-sys\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.598311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545f409e-df96-40d1-9915-d7e13edb4a3c-tmp\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.598311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598240    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vww2n\" (UniqueName: \"kubernetes.io/projected/83b7d0e1-6406-4ea2-bb44-8dc301131080-kube-api-access-vww2n\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.598311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-slash\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.598311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.598311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-cni-netd\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysctl-conf\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd26p\" (UniqueName: \"kubernetes.io/projected/e98185bd-0c41-43be-bb49-81fe4be59731-kube-api-access-cd26p\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-kubernetes\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e98185bd-0c41-43be-bb49-81fe4be59731-ovn-node-metrics-cert\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-tuned\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.598527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-kubelet\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-daemon-config\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-cni-bin\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-system-cni-dir\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-lib-modules\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-system-cni-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-system-cni-dir\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.598714    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-k8s-cni-cncf-io\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-netns\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.598798    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:06.098762834 +0000 UTC m=+2.009800102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-cni-bin\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0d0e19d-c65f-4d91-9151-8340d44d2502-serviceca\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.598960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhvx\" (UniqueName: \"kubernetes.io/projected/c0d0e19d-c65f-4d91-9151-8340d44d2502-kube-api-access-qjhvx\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-cni-multus\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.598999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-etc-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-node-log\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-ovnkube-script-lib\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7738d02-740f-4e53-9c3d-bcce706af7c7-agent-certs\") pod \"konnectivity-agent-n7fzz\" (UID: \"b7738d02-740f-4e53-9c3d-bcce706af7c7\") " pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-etc-kubernetes\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-systemd\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-ovn\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0d0e19d-c65f-4d91-9151-8340d44d2502-serviceca\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-sys-fs\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b390050-29b1-4801-891a-7dcd21d92725-iptables-alerter-script\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-hosts-file\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-run-netns\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599645    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-socket-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.599862 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599627    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgl7g\" (UniqueName: \"kubernetes.io/projected/80f1352e-1156-4d20-b03d-d6bb464633ee-kube-api-access-jgl7g\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-hosts-file\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7738d02-740f-4e53-9c3d-bcce706af7c7-konnectivity-ca\") pod \"konnectivity-agent-n7fzz\" (UID: \"b7738d02-740f-4e53-9c3d-bcce706af7c7\") " pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-run\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-cni-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-cnibin\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-hostroot\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-systemd-units\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-env-overrides\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-cni-binary-copy\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66mq\" (UniqueName: \"kubernetes.io/projected/9b390050-29b1-4801-891a-7dcd21d92725-kube-api-access-z66mq\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-systemd\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-socket-dir-parent\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.599997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-var-lib-kubelet\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.600718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600043    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0d0e19d-c65f-4d91-9151-8340d44d2502-host\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b390050-29b1-4801-891a-7dcd21d92725-iptables-alerter-script\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysconfig\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0d0e19d-c65f-4d91-9151-8340d44d2502-host\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkdb\" (UniqueName: \"kubernetes.io/projected/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-kube-api-access-dqkdb\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwtc\" (UniqueName: \"kubernetes.io/projected/9171816d-e76b-42c2-bb56-e341aaed3db4-kube-api-access-tdwtc\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7738d02-740f-4e53-9c3d-bcce706af7c7-konnectivity-ca\") pod \"konnectivity-agent-n7fzz\" (UID: \"b7738d02-740f-4e53-9c3d-bcce706af7c7\") " pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/159d4776-28fe-4b94-8cc4-85a239b5a5ba-kubelet-config\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-multus-certs\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-registration-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysctl-d\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-host\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-kubelet\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-device-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.601451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2bde71e-aa70-43ef-8b80-61755a1e7085-cni-binary-copy\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/159d4776-28fe-4b94-8cc4-85a239b5a5ba-dbus\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-conf-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-modprobe-d\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhpt\" (UniqueName: \"kubernetes.io/projected/a2bde71e-aa70-43ef-8b80-61755a1e7085-kube-api-access-dxhpt\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b390050-29b1-4801-891a-7dcd21d92725-host-slash\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-cnibin\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzvr\" (UniqueName: \"kubernetes.io/projected/545f409e-df96-40d1-9915-d7e13edb4a3c-kube-api-access-2dzvr\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b390050-29b1-4801-891a-7dcd21d92725-host-slash\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83b7d0e1-6406-4ea2-bb44-8dc301131080-cni-binary-copy\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600782    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-var-lib-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2bde71e-aa70-43ef-8b80-61755a1e7085-cnibin\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-log-socket\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-ovnkube-config\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.600867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-tmp-dir\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.602097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.601164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-tmp-dir\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.603049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.603029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7738d02-740f-4e53-9c3d-bcce706af7c7-agent-certs\") pod \"konnectivity-agent-n7fzz\" (UID: \"b7738d02-740f-4e53-9c3d-bcce706af7c7\") " pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.612303 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.612282    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:05.612303 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.612300    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:05.612417 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.612309    2565 projected.go:196] Error preparing data for projected volume kube-api-access-92bjl for pod openshift-network-diagnostics/network-check-target-ppwj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:05.612417 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.612360    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl podName:f705213e-23d4-458f-a7a8-ff4d4c70b21c nodeName:}" failed. No retries permitted until 2026-09-10 20:36:06.112349459 +0000 UTC m=+2.023386722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-92bjl" (UniqueName: "kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl") pod "network-check-target-ppwj5" (UID: "f705213e-23d4-458f-a7a8-ff4d4c70b21c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:05.621331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.621308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66mq\" (UniqueName: \"kubernetes.io/projected/9b390050-29b1-4801-891a-7dcd21d92725-kube-api-access-z66mq\") pod \"iptables-alerter-6s6vq\" (UID: \"9b390050-29b1-4801-891a-7dcd21d92725\") " pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.621463 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.621446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkdb\" (UniqueName: \"kubernetes.io/projected/5b91ad48-6bac-4b33-964b-f6cc1c5bada9-kube-api-access-dqkdb\") pod \"node-resolver-vzjhd\" (UID: \"5b91ad48-6bac-4b33-964b-f6cc1c5bada9\") " pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.621903 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.621887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwtc\" (UniqueName: \"kubernetes.io/projected/9171816d-e76b-42c2-bb56-e341aaed3db4-kube-api-access-tdwtc\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:05.622904 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.622889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhpt\" (UniqueName: \"kubernetes.io/projected/a2bde71e-aa70-43ef-8b80-61755a1e7085-kube-api-access-dxhpt\") pod \"multus-additional-cni-plugins-6nvt4\" (UID: \"a2bde71e-aa70-43ef-8b80-61755a1e7085\") " pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.623184 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.623016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhvx\" (UniqueName: \"kubernetes.io/projected/c0d0e19d-c65f-4d91-9151-8340d44d2502-kube-api-access-qjhvx\") pod \"node-ca-glc9l\" (UID: \"c0d0e19d-c65f-4d91-9151-8340d44d2502\") " pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.701473 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzvr\" (UniqueName: \"kubernetes.io/projected/545f409e-df96-40d1-9915-d7e13edb4a3c-kube-api-access-2dzvr\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.701579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83b7d0e1-6406-4ea2-bb44-8dc301131080-cni-binary-copy\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.701579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-var-lib-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-log-socket\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-var-lib-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701732 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-log-socket\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701732 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-ovnkube-config\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701732 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.701732 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-sys\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545f409e-df96-40d1-9915-d7e13edb4a3c-tmp\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vww2n\" (UniqueName: \"kubernetes.io/projected/83b7d0e1-6406-4ea2-bb44-8dc301131080-kube-api-access-vww2n\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-slash\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701809    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-sys\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701855    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.701912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-slash\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-cni-netd\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysctl-conf\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.701981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd26p\" (UniqueName: \"kubernetes.io/projected/e98185bd-0c41-43be-bb49-81fe4be59731-kube-api-access-cd26p\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-cni-netd\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-kubernetes\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e98185bd-0c41-43be-bb49-81fe4be59731-ovn-node-metrics-cert\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-tuned\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83b7d0e1-6406-4ea2-bb44-8dc301131080-cni-binary-copy\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-kubernetes\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-kubelet\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702122    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysctl-conf\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-kubelet\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-daemon-config\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-ovnkube-config\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-cni-bin\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-cni-bin\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-lib-modules\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.702331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-system-cni-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-lib-modules\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-k8s-cni-cncf-io\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-netns\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-cni-bin\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.702373    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-cni-multus\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-etc-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:05.702423    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret podName:159d4776-28fe-4b94-8cc4-85a239b5a5ba nodeName:}" failed. No retries permitted until 2026-09-10 20:36:06.202408209 +0000 UTC m=+2.113445473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret") pod "global-pull-secret-syncer-twjd6" (UID: "159d4776-28fe-4b94-8cc4-85a239b5a5ba") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-k8s-cni-cncf-io\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-etc-openvswitch\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-node-log\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-ovnkube-script-lib\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-cni-multus\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-etc-kubernetes\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702533    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-system-cni-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.703250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-systemd\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-ovn\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-var-lib-cni-bin\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-sys-fs\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-run-netns\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-systemd\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-daemon-config\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-socket-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-run-ovn\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-node-log\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-etc-kubernetes\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-sys-fs\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-run-netns\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-netns\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-socket-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl7g\" (UniqueName: \"kubernetes.io/projected/80f1352e-1156-4d20-b03d-d6bb464633ee-kube-api-access-jgl7g\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-run\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-cni-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-cnibin\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-hostroot\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-systemd-units\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.702999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-systemd-units\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-run\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-cni-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-env-overrides\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-ovnkube-script-lib\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-cnibin\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-hostroot\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-systemd\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-socket-dir-parent\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-var-lib-kubelet\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-systemd\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-socket-dir-parent\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysconfig\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysconfig\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.704836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-var-lib-kubelet\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/159d4776-28fe-4b94-8cc4-85a239b5a5ba-kubelet-config\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-multus-certs\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/159d4776-28fe-4b94-8cc4-85a239b5a5ba-kubelet-config\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-host-run-multus-certs\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-registration-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysctl-d\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-registration-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-host\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-kubelet\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-sysctl-d\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-host\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.705409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98185bd-0c41-43be-bb49-81fe4be59731-host-kubelet\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e98185bd-0c41-43be-bb49-81fe4be59731-env-overrides\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-device-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f1352e-1156-4d20-b03d-d6bb464633ee-device-dir\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/159d4776-28fe-4b94-8cc4-85a239b5a5ba-dbus\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-conf-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83b7d0e1-6406-4ea2-bb44-8dc301131080-multus-conf-dir\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/159d4776-28fe-4b94-8cc4-85a239b5a5ba-dbus\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.703911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-modprobe-d\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.704034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-modprobe-d\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.704080    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854af0dc85e6ffa23f95a26bb14f0ae7.slice/crio-01b2408358a059aa46b6e6ea2463e11ffd575efe34cd4beec8d9d36d40e35b63 WatchSource:0}: Error finding container 01b2408358a059aa46b6e6ea2463e11ffd575efe34cd4beec8d9d36d40e35b63: Status 404 returned error can't find the container with id 01b2408358a059aa46b6e6ea2463e11ffd575efe34cd4beec8d9d36d40e35b63
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.704784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545f409e-df96-40d1-9915-d7e13edb4a3c-etc-tuned\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.704879    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab19bc1a57527adb982141bacf53b44.slice/crio-a85c2ebbb33dd4963b260396ea0b9772214c6d676a679eacdcaa0880bacc658b WatchSource:0}: Error finding container a85c2ebbb33dd4963b260396ea0b9772214c6d676a679eacdcaa0880bacc658b: Status 404 returned error can't find the container with id a85c2ebbb33dd4963b260396ea0b9772214c6d676a679eacdcaa0880bacc658b
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.705743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e98185bd-0c41-43be-bb49-81fe4be59731-ovn-node-metrics-cert\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.706015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.705902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545f409e-df96-40d1-9915-d7e13edb4a3c-tmp\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.710434 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.710418    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:36:05.711287 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.711270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzvr\" (UniqueName: \"kubernetes.io/projected/545f409e-df96-40d1-9915-d7e13edb4a3c-kube-api-access-2dzvr\") pod \"tuned-twhdb\" (UID: \"545f409e-df96-40d1-9915-d7e13edb4a3c\") " pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.714823 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.714794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vww2n\" (UniqueName: \"kubernetes.io/projected/83b7d0e1-6406-4ea2-bb44-8dc301131080-kube-api-access-vww2n\") pod \"multus-9f7gx\" (UID: \"83b7d0e1-6406-4ea2-bb44-8dc301131080\") " pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.716274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.715295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd26p\" (UniqueName: \"kubernetes.io/projected/e98185bd-0c41-43be-bb49-81fe4be59731-kube-api-access-cd26p\") pod \"ovnkube-node-jwj4x\" (UID: \"e98185bd-0c41-43be-bb49-81fe4be59731\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.716274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.715576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl7g\" (UniqueName: \"kubernetes.io/projected/80f1352e-1156-4d20-b03d-d6bb464633ee-kube-api-access-jgl7g\") pod \"aws-ebs-csi-driver-node-kq6mc\" (UID: \"80f1352e-1156-4d20-b03d-d6bb464633ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.818892 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.818830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6s6vq"
Sep 10 20:36:05.827957 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.827933    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b390050_29b1_4801_891a_7dcd21d92725.slice/crio-4eb88db9764b8868e26144d295923630bcccb9e2336865bb41a1e152973c9f44 WatchSource:0}: Error finding container 4eb88db9764b8868e26144d295923630bcccb9e2336865bb41a1e152973c9f44: Status 404 returned error can't find the container with id 4eb88db9764b8868e26144d295923630bcccb9e2336865bb41a1e152973c9f44
Sep 10 20:36:05.836087 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.836072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vzjhd"
Sep 10 20:36:05.843440 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.843421    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b91ad48_6bac_4b33_964b_f6cc1c5bada9.slice/crio-7ccdb01a79756c548d983a4cd4cb7c76bf340773bec408e851ba4951a3a29221 WatchSource:0}: Error finding container 7ccdb01a79756c548d983a4cd4cb7c76bf340773bec408e851ba4951a3a29221: Status 404 returned error can't find the container with id 7ccdb01a79756c548d983a4cd4cb7c76bf340773bec408e851ba4951a3a29221
Sep 10 20:36:05.856624 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.856607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-glc9l"
Sep 10 20:36:05.858779 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.858763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6nvt4"
Sep 10 20:36:05.865210 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.865190    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d0e19d_c65f_4d91_9151_8340d44d2502.slice/crio-e34936b5120e915fed4d1d24b4a30c80466458fce85e8ee7461873be429d3365 WatchSource:0}: Error finding container e34936b5120e915fed4d1d24b4a30c80466458fce85e8ee7461873be429d3365: Status 404 returned error can't find the container with id e34936b5120e915fed4d1d24b4a30c80466458fce85e8ee7461873be429d3365
Sep 10 20:36:05.868025 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.867996    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bde71e_aa70_43ef_8b80_61755a1e7085.slice/crio-0c6c3dc3f55ae53f4e71bf6241b11a03374c04882b896ce481a7cc786710c00a WatchSource:0}: Error finding container 0c6c3dc3f55ae53f4e71bf6241b11a03374c04882b896ce481a7cc786710c00a: Status 404 returned error can't find the container with id 0c6c3dc3f55ae53f4e71bf6241b11a03374c04882b896ce481a7cc786710c00a
Sep 10 20:36:05.873554 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.873538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:05.878977 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.878958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:05.883282 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.883264    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7738d02_740f_4e53_9c3d_bcce706af7c7.slice/crio-9d7d69c788ce8df9158b185b525d9354d3f36405bdad34f71746822f3b10ab01 WatchSource:0}: Error finding container 9d7d69c788ce8df9158b185b525d9354d3f36405bdad34f71746822f3b10ab01: Status 404 returned error can't find the container with id 9d7d69c788ce8df9158b185b525d9354d3f36405bdad34f71746822f3b10ab01
Sep 10 20:36:05.888503 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.888486    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98185bd_0c41_43be_bb49_81fe4be59731.slice/crio-205581fc60f975d999eeb05d16c5217fa5b62b832d73ed39129420df03e82bb7 WatchSource:0}: Error finding container 205581fc60f975d999eeb05d16c5217fa5b62b832d73ed39129420df03e82bb7: Status 404 returned error can't find the container with id 205581fc60f975d999eeb05d16c5217fa5b62b832d73ed39129420df03e82bb7
Sep 10 20:36:05.902148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.902132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-twhdb"
Sep 10 20:36:05.910013 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.909866    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545f409e_df96_40d1_9915_d7e13edb4a3c.slice/crio-382f2e2041f78275ac38d22064e181907c7c47b653aaa0b2218c7fb9b217253d WatchSource:0}: Error finding container 382f2e2041f78275ac38d22064e181907c7c47b653aaa0b2218c7fb9b217253d: Status 404 returned error can't find the container with id 382f2e2041f78275ac38d22064e181907c7c47b653aaa0b2218c7fb9b217253d
Sep 10 20:36:05.933245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.933229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc"
Sep 10 20:36:05.936618 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:05.936600    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9f7gx"
Sep 10 20:36:05.942402 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.942382    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f1352e_1156_4d20_b03d_d6bb464633ee.slice/crio-31b351cedcf5e7fe8876e4e44a3483546f5b9a445ca2bda079776ac40445aedb WatchSource:0}: Error finding container 31b351cedcf5e7fe8876e4e44a3483546f5b9a445ca2bda079776ac40445aedb: Status 404 returned error can't find the container with id 31b351cedcf5e7fe8876e4e44a3483546f5b9a445ca2bda079776ac40445aedb
Sep 10 20:36:05.946308 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:05.946291    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b7d0e1_6406_4ea2_bb44_8dc301131080.slice/crio-cced14b539e53850e4e2b8fb38906d19ca6a2046c3c96609838f4c17b87fc48d WatchSource:0}: Error finding container cced14b539e53850e4e2b8fb38906d19ca6a2046c3c96609838f4c17b87fc48d: Status 404 returned error can't find the container with id cced14b539e53850e4e2b8fb38906d19ca6a2046c3c96609838f4c17b87fc48d
Sep 10 20:36:06.107043 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.107014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:06.107250 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.107151    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:06.107250 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.107215    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:07.107196337 +0000 UTC m=+3.018233602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:06.207749 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.207715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:06.207906 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.207782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:06.207969 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.207925    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:06.208022 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.207981    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret podName:159d4776-28fe-4b94-8cc4-85a239b5a5ba nodeName:}" failed. No retries permitted until 2026-09-10 20:36:07.207964167 +0000 UTC m=+3.119001435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret") pod "global-pull-secret-syncer-twjd6" (UID: "159d4776-28fe-4b94-8cc4-85a239b5a5ba") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:06.208337 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.208314    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:06.208393 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.208346    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:06.208393 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.208360    2565 projected.go:196] Error preparing data for projected volume kube-api-access-92bjl for pod openshift-network-diagnostics/network-check-target-ppwj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:06.208497 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:06.208414    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl podName:f705213e-23d4-458f-a7a8-ff4d4c70b21c nodeName:}" failed. No retries permitted until 2026-09-10 20:36:07.20839629 +0000 UTC m=+3.119433554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-92bjl" (UniqueName: "kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl") pod "network-check-target-ppwj5" (UID: "f705213e-23d4-458f-a7a8-ff4d4c70b21c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:06.413587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.413408    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:06.563207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.563140    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:31:05 +0000 UTC" deadline="2028-06-15 03:03:23.57476197 +0000 UTC"
Sep 10 20:36:06.563207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.563206    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15438h27m17.011559751s"
Sep 10 20:36:06.612743 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.612675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-twhdb" event={"ID":"545f409e-df96-40d1-9915-d7e13edb4a3c","Type":"ContainerStarted","Data":"382f2e2041f78275ac38d22064e181907c7c47b653aaa0b2218c7fb9b217253d"}
Sep 10 20:36:06.626863 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.626830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"205581fc60f975d999eeb05d16c5217fa5b62b832d73ed39129420df03e82bb7"}
Sep 10 20:36:06.639286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.639252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n7fzz" event={"ID":"b7738d02-740f-4e53-9c3d-bcce706af7c7","Type":"ContainerStarted","Data":"9d7d69c788ce8df9158b185b525d9354d3f36405bdad34f71746822f3b10ab01"}
Sep 10 20:36:06.664146 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.664079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerStarted","Data":"0c6c3dc3f55ae53f4e71bf6241b11a03374c04882b896ce481a7cc786710c00a"}
Sep 10 20:36:06.677918 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.677890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-glc9l" event={"ID":"c0d0e19d-c65f-4d91-9151-8340d44d2502","Type":"ContainerStarted","Data":"e34936b5120e915fed4d1d24b4a30c80466458fce85e8ee7461873be429d3365"}
Sep 10 20:36:06.680516 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.680491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vzjhd" event={"ID":"5b91ad48-6bac-4b33-964b-f6cc1c5bada9","Type":"ContainerStarted","Data":"7ccdb01a79756c548d983a4cd4cb7c76bf340773bec408e851ba4951a3a29221"}
Sep 10 20:36:06.683505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.683479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9f7gx" event={"ID":"83b7d0e1-6406-4ea2-bb44-8dc301131080","Type":"ContainerStarted","Data":"cced14b539e53850e4e2b8fb38906d19ca6a2046c3c96609838f4c17b87fc48d"}
Sep 10 20:36:06.687771 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.687678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc" event={"ID":"80f1352e-1156-4d20-b03d-d6bb464633ee","Type":"ContainerStarted","Data":"31b351cedcf5e7fe8876e4e44a3483546f5b9a445ca2bda079776ac40445aedb"}
Sep 10 20:36:06.694115 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.694053    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6s6vq" event={"ID":"9b390050-29b1-4801-891a-7dcd21d92725","Type":"ContainerStarted","Data":"4eb88db9764b8868e26144d295923630bcccb9e2336865bb41a1e152973c9f44"}
Sep 10 20:36:06.709225 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.709197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" event={"ID":"854af0dc85e6ffa23f95a26bb14f0ae7","Type":"ContainerStarted","Data":"01b2408358a059aa46b6e6ea2463e11ffd575efe34cd4beec8d9d36d40e35b63"}
Sep 10 20:36:06.717231 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:06.717207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal" event={"ID":"2ab19bc1a57527adb982141bacf53b44","Type":"ContainerStarted","Data":"a85c2ebbb33dd4963b260396ea0b9772214c6d676a679eacdcaa0880bacc658b"}
Sep 10 20:36:07.102369 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.102339    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:07.114223 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.114129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:07.114358 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.114312    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:07.114424 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.114365    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:09.114348661 +0000 UTC m=+5.025385930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:07.214702 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.214660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:07.214860 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.214753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:07.214923 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.214901    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:07.214923 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.214918    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:07.215019 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.214930    2565 projected.go:196] Error preparing data for projected volume kube-api-access-92bjl for pod openshift-network-diagnostics/network-check-target-ppwj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:07.215019 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.214983    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl podName:f705213e-23d4-458f-a7a8-ff4d4c70b21c nodeName:}" failed. No retries permitted until 2026-09-10 20:36:09.214965553 +0000 UTC m=+5.126002828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-92bjl" (UniqueName: "kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl") pod "network-check-target-ppwj5" (UID: "f705213e-23d4-458f-a7a8-ff4d4c70b21c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:07.215133 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.215050    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:07.215133 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.215082    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret podName:159d4776-28fe-4b94-8cc4-85a239b5a5ba nodeName:}" failed. No retries permitted until 2026-09-10 20:36:09.215071007 +0000 UTC m=+5.126108274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret") pod "global-pull-secret-syncer-twjd6" (UID: "159d4776-28fe-4b94-8cc4-85a239b5a5ba") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:07.564147 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.563930    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:31:05 +0000 UTC" deadline="2028-03-12 14:25:41.491101632 +0000 UTC"
Sep 10 20:36:07.564147 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.563992    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13169h49m33.927114592s"
Sep 10 20:36:07.584337 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.584307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:07.584484 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.584437    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:07.584884 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.584864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:07.585007 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.584979    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:07.585068 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:07.585045    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:07.585128 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:07.585112    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:09.130322 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:09.130250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:09.130778 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.130411    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:09.130778 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.130467    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:13.130449789 +0000 UTC m=+9.041487056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:09.231222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:09.231283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.231395    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.231451    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret podName:159d4776-28fe-4b94-8cc4-85a239b5a5ba nodeName:}" failed. No retries permitted until 2026-09-10 20:36:13.231430602 +0000 UTC m=+9.142467869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret") pod "global-pull-secret-syncer-twjd6" (UID: "159d4776-28fe-4b94-8cc4-85a239b5a5ba") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.231499    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.231516    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.231527    2565 projected.go:196] Error preparing data for projected volume kube-api-access-92bjl for pod openshift-network-diagnostics/network-check-target-ppwj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:09.231623 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.231564    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl podName:f705213e-23d4-458f-a7a8-ff4d4c70b21c nodeName:}" failed. No retries permitted until 2026-09-10 20:36:13.231552001 +0000 UTC m=+9.142589278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-92bjl" (UniqueName: "kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl") pod "network-check-target-ppwj5" (UID: "f705213e-23d4-458f-a7a8-ff4d4c70b21c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:09.585896 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:09.584702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:09.585896 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.584833    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:09.585896 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:09.584894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:09.585896 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.584971    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:09.585896 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:09.585008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:09.585896 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:09.585068    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:11.584603 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:11.584162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:11.584603 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:11.584292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:11.584603 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:11.584293    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:11.584603 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:11.584418    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:11.584603 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:11.584454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:11.584603 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:11.584521    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:13.164599 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:13.164564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:13.165087 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.164745    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:13.165087 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.164821    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:21.164798137 +0000 UTC m=+17.075835405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:13.265914 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:13.265886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:13.266077 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:13.265940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:13.266077 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.266054    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:13.266077 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.266058    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:13.266196 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.266079    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:13.266196 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.266091    2565 projected.go:196] Error preparing data for projected volume kube-api-access-92bjl for pod openshift-network-diagnostics/network-check-target-ppwj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:13.266196 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.266116    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret podName:159d4776-28fe-4b94-8cc4-85a239b5a5ba nodeName:}" failed. No retries permitted until 2026-09-10 20:36:21.26609716 +0000 UTC m=+17.177134425 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret") pod "global-pull-secret-syncer-twjd6" (UID: "159d4776-28fe-4b94-8cc4-85a239b5a5ba") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:13.266196 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.266133    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl podName:f705213e-23d4-458f-a7a8-ff4d4c70b21c nodeName:}" failed. No retries permitted until 2026-09-10 20:36:21.266124326 +0000 UTC m=+17.177161593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-92bjl" (UniqueName: "kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl") pod "network-check-target-ppwj5" (UID: "f705213e-23d4-458f-a7a8-ff4d4c70b21c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:13.584712 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:13.584666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:13.584900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:13.584803    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:13.584900 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.584816    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:13.585019 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.584929    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:13.585019 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:13.584964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:13.585122 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:13.585060    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:15.584475 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:15.584441    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:15.584988 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:15.584461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:15.584988 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:15.584561    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:15.584988 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:15.584578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:15.584988 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:15.584702    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:15.584988 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:15.584790    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:17.584475 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:17.584442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:17.584961 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:17.584443    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:17.584961 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:17.584565    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:17.584961 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:17.584643    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:17.584961 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:17.584442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:17.584961 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:17.584785    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:19.584431 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:19.584397    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:19.584431 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:19.584426    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:19.584912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:19.584449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:19.584912 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:19.584521    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:19.584912 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:19.584660    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:19.584912 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:19.584823    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:21.225170 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:21.225136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:21.225650 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.225284    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:21.225650 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.225350    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:37.225335369 +0000 UTC m=+33.136372631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:21.325723 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:21.325678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:21.325956 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:21.325748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:21.325956 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.325852    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:21.325956 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.325860    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:21.325956 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.325888    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:21.325956 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.325900    2565 projected.go:196] Error preparing data for projected volume kube-api-access-92bjl for pod openshift-network-diagnostics/network-check-target-ppwj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:21.325956 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.325910    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret podName:159d4776-28fe-4b94-8cc4-85a239b5a5ba nodeName:}" failed. No retries permitted until 2026-09-10 20:36:37.325891839 +0000 UTC m=+33.236929125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret") pod "global-pull-secret-syncer-twjd6" (UID: "159d4776-28fe-4b94-8cc4-85a239b5a5ba") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:21.325956 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.325946    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl podName:f705213e-23d4-458f-a7a8-ff4d4c70b21c nodeName:}" failed. No retries permitted until 2026-09-10 20:36:37.325930724 +0000 UTC m=+33.236968009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-92bjl" (UniqueName: "kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl") pod "network-check-target-ppwj5" (UID: "f705213e-23d4-458f-a7a8-ff4d4c70b21c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:21.584172 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:21.584096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:21.584329 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:21.584100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:21.584329 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.584198    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:21.584329 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:21.584100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:21.584494 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.584282    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:21.584494 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:21.584393    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:23.585050 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:23.585014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:23.585050 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:23.585014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:23.585502 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:23.585115    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:23.585502 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:23.585026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:23.585502 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:23.585211    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:23.585502 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:23.585285    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:24.765240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:24.764936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-twhdb" event={"ID":"545f409e-df96-40d1-9915-d7e13edb4a3c","Type":"ContainerStarted","Data":"e9500880f938bdaef637fa3047fb58859c2c6493aca74a33a42cda81d4fa0eb2"}
Sep 10 20:36:24.767023 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:24.766829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"01489bd4ba3cbf7a030f89b25dcba327df18ce858ff19c89f11cdb8ac8a50bc5"}
Sep 10 20:36:24.768805 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:24.768719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9f7gx" event={"ID":"83b7d0e1-6406-4ea2-bb44-8dc301131080","Type":"ContainerStarted","Data":"dc337e1a0ebb838af964fb1c55fa2abfe2821ce27f7e08c07def51135ed38196"}
Sep 10 20:36:24.771008 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:24.770973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal" event={"ID":"2ab19bc1a57527adb982141bacf53b44","Type":"ContainerStarted","Data":"f664c6039e770200765fd4b5f0348c797f146336de06ad5b4134852582fa355a"}
Sep 10 20:36:24.783741 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:24.783704    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-twhdb" podStartSLOduration=2.135535494 podStartE2EDuration="20.783675517s" totalImagesPullingTime="18.648140023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:24.783383086 +0000 UTC m=+20.694420370" watchObservedRunningTime="2026-09-10 20:36:24.783675517 +0000 UTC m=+20.694712800"
Sep 10 20:36:24.799160 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:24.799125    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal" podStartSLOduration=19.799111621 podStartE2EDuration="19.799111621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:24.798958328 +0000 UTC m=+20.709995614" watchObservedRunningTime="2026-09-10 20:36:24.799111621 +0000 UTC m=+20.710148905"
Sep 10 20:36:25.584419 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.584138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:25.584567 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.584152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:25.584567 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:25.584488    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:25.584567 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.584207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:25.584567 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:25.584540    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:25.584799 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:25.584599    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:25.776884 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.776857    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:36:25.777439 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.777279    2565 generic.go:356] "Generic (PLEG): container finished" podID="e98185bd-0c41-43be-bb49-81fe4be59731" containerID="1a4c83b664b29f4ba9453db52876f477603a6f989c613fbebe7b6b9185d0e66b" exitCode=1
Sep 10 20:36:25.777439 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.777364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"cea200eb98fc347957162d043003312493f1b77f361d07d795b8ef0ebf9064c7"}
Sep 10 20:36:25.777544 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.777435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"18e97ca3fe1d12bcc3dfe7703b5e929aefaa8c14bc372e266a612a708f9154a1"}
Sep 10 20:36:25.777544 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.777486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"dfcbcee053764c2085f6fd25712dce36b657755e2cfa1f88005f6d3e24b1d95c"}
Sep 10 20:36:25.777544 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.777534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"f5ae9d6c9f309e8d41bf23808dc3c4a280fd7a21ed4b28ba4b223868bd6e73ad"}
Sep 10 20:36:25.777707 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.777579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerDied","Data":"1a4c83b664b29f4ba9453db52876f477603a6f989c613fbebe7b6b9185d0e66b"}
Sep 10 20:36:25.778887 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.778855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n7fzz" event={"ID":"b7738d02-740f-4e53-9c3d-bcce706af7c7","Type":"ContainerStarted","Data":"deb98b2bf47a5cb31fe4d2f614c5f13e7d5d7ff76d8d2c21df0d7647e6c19213"}
Sep 10 20:36:25.780538 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.780512    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2bde71e-aa70-43ef-8b80-61755a1e7085" containerID="d393e6def13ba780b422e0a22d3093c9fa5566dbebc51eced4f13ccb84e7ce80" exitCode=0
Sep 10 20:36:25.780632 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.780577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerDied","Data":"d393e6def13ba780b422e0a22d3093c9fa5566dbebc51eced4f13ccb84e7ce80"}
Sep 10 20:36:25.782188 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.782158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-glc9l" event={"ID":"c0d0e19d-c65f-4d91-9151-8340d44d2502","Type":"ContainerStarted","Data":"351fe7677edee662dee3ec7df15f08dc6290b928eabafc2c45b5af3ad075680c"}
Sep 10 20:36:25.784669 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.784639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vzjhd" event={"ID":"5b91ad48-6bac-4b33-964b-f6cc1c5bada9","Type":"ContainerStarted","Data":"760f17a53d5c7b90a3cadfe59d97740e18df4ca82c65362b20db6216060c2049"}
Sep 10 20:36:25.786846 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.786672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc" event={"ID":"80f1352e-1156-4d20-b03d-d6bb464633ee","Type":"ContainerStarted","Data":"e9c5de8ec2ae4b677d437fd179640d7ced2028008af805a1e8a5a18f27cb4412"}
Sep 10 20:36:25.788937 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.788917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6s6vq" event={"ID":"9b390050-29b1-4801-891a-7dcd21d92725","Type":"ContainerStarted","Data":"97a724203a059df51a86320a427fe8ff68ac5c891dc2b3360f59a1a84c5a144e"}
Sep 10 20:36:25.790548 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.790526    2565 generic.go:356] "Generic (PLEG): container finished" podID="854af0dc85e6ffa23f95a26bb14f0ae7" containerID="d32bdffda7bb7a49a8a7b058da7b9710cb7fd0ef30d49f1d5d916549c74bc906" exitCode=0
Sep 10 20:36:25.790642 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.790607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" event={"ID":"854af0dc85e6ffa23f95a26bb14f0ae7","Type":"ContainerDied","Data":"d32bdffda7bb7a49a8a7b058da7b9710cb7fd0ef30d49f1d5d916549c74bc906"}
Sep 10 20:36:25.820948 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.820901    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n7fzz" podStartSLOduration=3.173221645 podStartE2EDuration="21.820884321s" totalImagesPullingTime="18.647662676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:25.800798312 +0000 UTC m=+21.711835596" watchObservedRunningTime="2026-09-10 20:36:25.820884321 +0000 UTC m=+21.731921609"
Sep 10 20:36:25.821576 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.821547    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vzjhd" podStartSLOduration=3.154314751 podStartE2EDuration="21.821537936s" totalImagesPullingTime="18.667223185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:25.821023454 +0000 UTC m=+21.732060729" watchObservedRunningTime="2026-09-10 20:36:25.821537936 +0000 UTC m=+21.732575221"
Sep 10 20:36:25.860434 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.860009    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-glc9l" podStartSLOduration=3.194842487 podStartE2EDuration="21.859994573s" totalImagesPullingTime="18.665152086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:25.859704632 +0000 UTC m=+21.770741913" watchObservedRunningTime="2026-09-10 20:36:25.859994573 +0000 UTC m=+21.771031859"
Sep 10 20:36:25.907221 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:25.907185    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9f7gx" podStartSLOduration=3.254649712 podStartE2EDuration="21.907171659s" totalImagesPullingTime="18.652521947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:25.906750162 +0000 UTC m=+21.817787449" watchObservedRunningTime="2026-09-10 20:36:25.907171659 +0000 UTC m=+21.818208970"
Sep 10 20:36:26.051209 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.051179    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:36:26.567275 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.567173    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:36:26.051199472Z","UUID":"a641122c-820e-45e0-a418-4650b5668dfd","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:36:26.569020 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.568997    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:36:26.569166 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.569031    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:36:26.796107 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.796070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc" event={"ID":"80f1352e-1156-4d20-b03d-d6bb464633ee","Type":"ContainerStarted","Data":"2c764e3674218e453ec7574c156f354f38f892d2622605515354d0cedc5ce2f1"}
Sep 10 20:36:26.798623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.798590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" event={"ID":"854af0dc85e6ffa23f95a26bb14f0ae7","Type":"ContainerStarted","Data":"5ff70f784cec4a9ad0a2d4c206b3ddec8e99213cdad02f5704ad20815df9f5bc"}
Sep 10 20:36:26.819647 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.819557    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6s6vq" podStartSLOduration=4.116711919 podStartE2EDuration="22.819539507s" totalImagesPullingTime="18.702827588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:25.923665841 +0000 UTC m=+21.834703124" watchObservedRunningTime="2026-09-10 20:36:26.819539507 +0000 UTC m=+22.730576790"
Sep 10 20:36:26.819858 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:26.819823    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" podStartSLOduration=21.819812308 podStartE2EDuration="21.819812308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:26.819539455 +0000 UTC m=+22.730576738" watchObservedRunningTime="2026-09-10 20:36:26.819812308 +0000 UTC m=+22.730849592"
Sep 10 20:36:27.584805 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:27.584575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:27.584976 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:27.584603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:27.584976 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:27.584835    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:27.584976 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:27.584929    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:27.584976 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:27.584606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:27.585154 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:27.585035    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:27.804414 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:27.804009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc" event={"ID":"80f1352e-1156-4d20-b03d-d6bb464633ee","Type":"ContainerStarted","Data":"85c8b9400de525fac16a9e8ab8377b4824ec9b009f83a725d1a0c6ef29dd2690"}
Sep 10 20:36:27.807797 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:27.807775    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:36:27.808262 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:27.808238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"7e3e65ba5c49ab3221ea85c0cb2097894a52db0ef1fcbd3258d4281aadf81976"}
Sep 10 20:36:27.824902 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:27.824861    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq6mc" podStartSLOduration=3.4812113350000002 podStartE2EDuration="23.824847953s" totalImagesPullingTime="20.343636618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:27.82455991 +0000 UTC m=+23.735597195" watchObservedRunningTime="2026-09-10 20:36:27.824847953 +0000 UTC m=+23.735885237"
Sep 10 20:36:29.584900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:29.584828    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:29.585358 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:29.584924    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:29.585358 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:29.584933    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:29.585358 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:29.584945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:29.585358 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:29.585033    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:29.585358 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:29.585128    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:29.662007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:29.661972    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:29.662543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:29.662517    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:30.818128 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:30.817981    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:36:30.818519 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:30.818488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"a4559bb83511d6be5af9b3cad3f5417253ae0a096ed5418103a529888617120c"}
Sep 10 20:36:30.818743 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:30.818724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:30.818973 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:30.818952    2565 scope.go:117] "RemoveContainer" containerID="1a4c83b664b29f4ba9453db52876f477603a6f989c613fbebe7b6b9185d0e66b"
Sep 10 20:36:30.820421 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:30.820393    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2bde71e-aa70-43ef-8b80-61755a1e7085" containerID="c4697e9403fe7fd4eeaf9f8811eceb5075bbf695838cf708adb1cff48d4200d5" exitCode=0
Sep 10 20:36:30.820500 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:30.820428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerDied","Data":"c4697e9403fe7fd4eeaf9f8811eceb5075bbf695838cf708adb1cff48d4200d5"}
Sep 10 20:36:30.835529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:30.835504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:31.584218 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.584190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:31.584426 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.584221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:31.584426 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.584191    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:31.584426 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:31.584342    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:31.584564 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:31.584420    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:31.584564 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:31.584495    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:31.829048 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.829026    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:36:31.829628 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.829383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" event={"ID":"e98185bd-0c41-43be-bb49-81fe4be59731","Type":"ContainerStarted","Data":"523cc226ccb6f5d6dec1672c97110c7023b9866567e2f48429fecb5cb8d3c006"}
Sep 10 20:36:31.829677 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.829631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:31.829818 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.829786    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:31.831808 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.831784    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2bde71e-aa70-43ef-8b80-61755a1e7085" containerID="7fdcccca7eaeef348467cb321d72873b8cdf30af567e826fc1fcc87c874d065f" exitCode=0
Sep 10 20:36:31.831898 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.831837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerDied","Data":"7fdcccca7eaeef348467cb321d72873b8cdf30af567e826fc1fcc87c874d065f"}
Sep 10 20:36:31.844834 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.844817    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:36:31.862920 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.862885    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x" podStartSLOduration=9.138116729 podStartE2EDuration="27.862873956s" totalImagesPullingTime="18.724757227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:31.862736758 +0000 UTC m=+27.773774033" watchObservedRunningTime="2026-09-10 20:36:31.862873956 +0000 UTC m=+27.773911240"
Sep 10 20:36:31.985795 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.985763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ppwj5"]
Sep 10 20:36:31.985963 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.985867    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:31.986027 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:31.985964    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:31.988292 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.988270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kjnmq"]
Sep 10 20:36:31.988399 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:31.988355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:31.988448 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:31.988425    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:32.005066 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:32.005044    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-twjd6"]
Sep 10 20:36:32.005161 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:32.005118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:32.005226 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:32.005195    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:32.836043 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:32.835948    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2bde71e-aa70-43ef-8b80-61755a1e7085" containerID="5b9587a409528e5d294f3a511584ffbcaa0c1a8a81feb32ee803748466f9b8eb" exitCode=0
Sep 10 20:36:32.836043 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:32.835995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerDied","Data":"5b9587a409528e5d294f3a511584ffbcaa0c1a8a81feb32ee803748466f9b8eb"}
Sep 10 20:36:33.584255 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:33.584218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:33.584414 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:33.584341    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:33.584414 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:33.584338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:33.584535 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:33.584344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:33.584535 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:33.584480    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:33.584639 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:33.584591    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:33.737872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:33.737619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:33.737996 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:33.737901    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 20:36:33.738197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:33.738177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n7fzz"
Sep 10 20:36:34.578488 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:34.578421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n7fzz_b7738d02-740f-4e53-9c3d-bcce706af7c7/konnectivity-agent/0.log" limit=1024
Sep 10 20:36:34.677442 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:34.677402    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n7fzz_b7738d02-740f-4e53-9c3d-bcce706af7c7/konnectivity-agent/0.log" limit=1024
Sep 10 20:36:35.584171 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:35.584141    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:35.584533 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:35.584234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:35.584533 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:35.584249    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twjd6" podUID="159d4776-28fe-4b94-8cc4-85a239b5a5ba"
Sep 10 20:36:35.584533 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:35.584319    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnmq" podUID="9171816d-e76b-42c2-bb56-e341aaed3db4"
Sep 10 20:36:35.584533 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:35.584357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:35.584533 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:35.584423    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppwj5" podUID="f705213e-23d4-458f-a7a8-ff4d4c70b21c"
Sep 10 20:36:37.249615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.249587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:37.250016 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.249798    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:37.250016 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.249900    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:09.249877001 +0000 UTC m=+65.160914276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:37.350302 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.350267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:37.350477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.350321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:37.350477 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.350422    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:37.350477 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.350471    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret podName:159d4776-28fe-4b94-8cc4-85a239b5a5ba nodeName:}" failed. No retries permitted until 2026-09-10 20:37:09.35045841 +0000 UTC m=+65.261495673 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret") pod "global-pull-secret-syncer-twjd6" (UID: "159d4776-28fe-4b94-8cc4-85a239b5a5ba") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:37.350641 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.350424    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:37.350641 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.350521    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:37.350641 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.350534    2565 projected.go:196] Error preparing data for projected volume kube-api-access-92bjl for pod openshift-network-diagnostics/network-check-target-ppwj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:37.350641 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.350581    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl podName:f705213e-23d4-458f-a7a8-ff4d4c70b21c nodeName:}" failed. No retries permitted until 2026-09-10 20:37:09.350569169 +0000 UTC m=+65.261606444 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-92bjl" (UniqueName: "kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl") pod "network-check-target-ppwj5" (UID: "f705213e-23d4-458f-a7a8-ff4d4c70b21c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:37.388796 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.388775    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeReady"
Sep 10 20:36:37.388935 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.388921    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:36:37.440304 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.440247    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hkl9m"]
Sep 10 20:36:37.462152 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.462131    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bp5xz"]
Sep 10 20:36:37.462293 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.462272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.465299 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.465280    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:36:37.465412 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.465281    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:36:37.465584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.465569    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-89pjq\""
Sep 10 20:36:37.474334 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.474315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hkl9m"]
Sep 10 20:36:37.474435 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.474347    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bp5xz"]
Sep 10 20:36:37.474435 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.474417    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:37.477553 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.477535    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2vtqg\""
Sep 10 20:36:37.477653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.477578    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:36:37.477653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.477540    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:36:37.477957 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.477942    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:36:37.551932 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.551906    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.552085 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.551954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm89f\" (UniqueName: \"kubernetes.io/projected/b492afb9-cddd-4ef5-a90e-6c89771312a9-kube-api-access-zm89f\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.552085 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.551977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b492afb9-cddd-4ef5-a90e-6c89771312a9-tmp-dir\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.552085 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.552006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ndz\" (UniqueName: \"kubernetes.io/projected/dbfc039d-4acd-4649-a477-b18f2c799a86-kube-api-access-44ndz\") pod \"ingress-canary-bp5xz\" (UID: \"dbfc039d-4acd-4649-a477-b18f2c799a86\") " pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:37.552213 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.552089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b492afb9-cddd-4ef5-a90e-6c89771312a9-config-volume\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.552213 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.552157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbfc039d-4acd-4649-a477-b18f2c799a86-cert\") pod \"ingress-canary-bp5xz\" (UID: \"dbfc039d-4acd-4649-a477-b18f2c799a86\") " pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:37.584292 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.584265    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:36:37.584411 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.584268    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:36:37.584561 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.584268    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:36:37.587700 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.587663    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:36:37.587810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.587732    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ks6z6\""
Sep 10 20:36:37.587865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.587668    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:36:37.587865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.587662    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:36:37.587956 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.587940    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:36:37.588114 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.588097    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24xx\""
Sep 10 20:36:37.652916 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.652883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b492afb9-cddd-4ef5-a90e-6c89771312a9-config-volume\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.653034 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.652939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbfc039d-4acd-4649-a477-b18f2c799a86-cert\") pod \"ingress-canary-bp5xz\" (UID: \"dbfc039d-4acd-4649-a477-b18f2c799a86\") " pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:37.653034 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.652991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.653034 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.653025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm89f\" (UniqueName: \"kubernetes.io/projected/b492afb9-cddd-4ef5-a90e-6c89771312a9-kube-api-access-zm89f\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.653184 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.653052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b492afb9-cddd-4ef5-a90e-6c89771312a9-tmp-dir\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.653184 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.653079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ndz\" (UniqueName: \"kubernetes.io/projected/dbfc039d-4acd-4649-a477-b18f2c799a86-kube-api-access-44ndz\") pod \"ingress-canary-bp5xz\" (UID: \"dbfc039d-4acd-4649-a477-b18f2c799a86\") " pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:37.653184 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.653157    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:37.653324 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:37.653260    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls podName:b492afb9-cddd-4ef5-a90e-6c89771312a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:38.153217881 +0000 UTC m=+34.064255165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls") pod "dns-default-hkl9m" (UID: "b492afb9-cddd-4ef5-a90e-6c89771312a9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:37.653434 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.653414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b492afb9-cddd-4ef5-a90e-6c89771312a9-tmp-dir\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.653559 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.653521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b492afb9-cddd-4ef5-a90e-6c89771312a9-config-volume\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.658512 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.658487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbfc039d-4acd-4649-a477-b18f2c799a86-cert\") pod \"ingress-canary-bp5xz\" (UID: \"dbfc039d-4acd-4649-a477-b18f2c799a86\") " pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:37.663371 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.663338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ndz\" (UniqueName: \"kubernetes.io/projected/dbfc039d-4acd-4649-a477-b18f2c799a86-kube-api-access-44ndz\") pod \"ingress-canary-bp5xz\" (UID: \"dbfc039d-4acd-4649-a477-b18f2c799a86\") " pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:37.663658 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.663634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm89f\" (UniqueName: \"kubernetes.io/projected/b492afb9-cddd-4ef5-a90e-6c89771312a9-kube-api-access-zm89f\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:37.783982 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:37.783912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bp5xz"
Sep 10 20:36:38.156085 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:38.156047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:38.156240 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:38.156189    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:38.156280 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:38.156245    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls podName:b492afb9-cddd-4ef5-a90e-6c89771312a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:39.156230222 +0000 UTC m=+35.067267483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls") pod "dns-default-hkl9m" (UID: "b492afb9-cddd-4ef5-a90e-6c89771312a9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:38.494255 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:38.494178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bp5xz"]
Sep 10 20:36:38.497492 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:36:38.497465    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfc039d_4acd_4649_a477_b18f2c799a86.slice/crio-ebc68e5571039794e683d920bcfee9139127b23d0e76334327d8e30afe2ffbdd WatchSource:0}: Error finding container ebc68e5571039794e683d920bcfee9139127b23d0e76334327d8e30afe2ffbdd: Status 404 returned error can't find the container with id ebc68e5571039794e683d920bcfee9139127b23d0e76334327d8e30afe2ffbdd
Sep 10 20:36:38.851542 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:38.851385    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2bde71e-aa70-43ef-8b80-61755a1e7085" containerID="b1f2c1e156645ad022b7d1ef68bb97ef65bec017c92189da13c7373f4b54af63" exitCode=0
Sep 10 20:36:38.851681 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:38.851463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerDied","Data":"b1f2c1e156645ad022b7d1ef68bb97ef65bec017c92189da13c7373f4b54af63"}
Sep 10 20:36:38.854267 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:38.854226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bp5xz" event={"ID":"dbfc039d-4acd-4649-a477-b18f2c799a86","Type":"ContainerStarted","Data":"ebc68e5571039794e683d920bcfee9139127b23d0e76334327d8e30afe2ffbdd"}
Sep 10 20:36:39.164801 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:39.164771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:39.164910 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:39.164885    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:39.164960 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:39.164937    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls podName:b492afb9-cddd-4ef5-a90e-6c89771312a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:41.164921543 +0000 UTC m=+37.075958808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls") pod "dns-default-hkl9m" (UID: "b492afb9-cddd-4ef5-a90e-6c89771312a9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:39.860298 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:39.860268    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2bde71e-aa70-43ef-8b80-61755a1e7085" containerID="46eb23a68f0ca25be7ad503583e1aeba58057020a380e4d49824ced6e886d763" exitCode=0
Sep 10 20:36:39.860845 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:39.860320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerDied","Data":"46eb23a68f0ca25be7ad503583e1aeba58057020a380e4d49824ced6e886d763"}
Sep 10 20:36:40.867293 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:40.867261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" event={"ID":"a2bde71e-aa70-43ef-8b80-61755a1e7085","Type":"ContainerStarted","Data":"af4e3c12644afdf13f3632a0fde72f996ab4154aa4ea60c50a8eae0ca9160b3b"}
Sep 10 20:36:40.896048 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:40.895999    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6nvt4" podStartSLOduration=6.879042586 podStartE2EDuration="36.895980502s" totalImagesPullingTime="30.016937916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:40.894355046 +0000 UTC m=+36.805392341" watchObservedRunningTime="2026-09-10 20:36:40.895980502 +0000 UTC m=+36.807017787"
Sep 10 20:36:41.180581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:41.180561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:41.180818 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:41.180742    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:41.180818 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:41.180806    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls podName:b492afb9-cddd-4ef5-a90e-6c89771312a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:45.180787406 +0000 UTC m=+41.091824669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls") pod "dns-default-hkl9m" (UID: "b492afb9-cddd-4ef5-a90e-6c89771312a9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:41.871164 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:41.871130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bp5xz" event={"ID":"dbfc039d-4acd-4649-a477-b18f2c799a86","Type":"ContainerStarted","Data":"c2b6600943bb7f79fc86e32678f2493bf2e0e80a6c1b1a5877a3a209ffa47c1c"}
Sep 10 20:36:41.894349 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:41.894299    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bp5xz" podStartSLOduration=2.225267986 podStartE2EDuration="4.894284523s" totalImagesPullingTime="2.669016537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:41.893392876 +0000 UTC m=+37.804430159" watchObservedRunningTime="2026-09-10 20:36:41.894284523 +0000 UTC m=+37.805321806"
Sep 10 20:36:45.206886 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:45.206854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:45.207239 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:45.206951    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:45.207239 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:45.206998    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls podName:b492afb9-cddd-4ef5-a90e-6c89771312a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:53.206985491 +0000 UTC m=+49.118022753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls") pod "dns-default-hkl9m" (UID: "b492afb9-cddd-4ef5-a90e-6c89771312a9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:53.259863 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:36:53.259824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:36:53.260343 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:53.259960    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:53.260343 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:36:53.260014    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls podName:b492afb9-cddd-4ef5-a90e-6c89771312a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:09.260000652 +0000 UTC m=+65.171037913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls") pod "dns-default-hkl9m" (UID: "b492afb9-cddd-4ef5-a90e-6c89771312a9") : secret "dns-default-metrics-tls" not found
Sep 10 20:37:03.849872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:03.849841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwj4x"
Sep 10 20:37:04.658665 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.658638    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-zz76z"]
Sep 10 20:37:04.660999 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.660986    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.664030 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.664004    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 20:37:04.664030 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.664028    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-c22z2\""
Sep 10 20:37:04.664208 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.664125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 20:37:04.665504 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.665484    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:37:04.665836 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.665809    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 20:37:04.671066 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.671047    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 20:37:04.675547 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.675527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-zz76z"]
Sep 10 20:37:04.832168 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.832144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxppv\" (UniqueName: \"kubernetes.io/projected/c0845dc6-da33-455e-9c95-4c9776458de3-kube-api-access-fxppv\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.832274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.832177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0845dc6-da33-455e-9c95-4c9776458de3-config\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.832274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.832193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0845dc6-da33-455e-9c95-4c9776458de3-serving-cert\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.832274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.832210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0845dc6-da33-455e-9c95-4c9776458de3-trusted-ca\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.863208 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.863181    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-w5d4j"]
Sep 10 20:37:04.865060 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.865042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:04.868981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.868962    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:37:04.869259 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.869244    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dw7tw\""
Sep 10 20:37:04.869348 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.869327    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 20:37:04.869348 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.869346    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 20:37:04.869649 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.869636    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:37:04.875334 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.875314    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"]
Sep 10 20:37:04.877680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.877659    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55db58887f-8zf5r"]
Sep 10 20:37:04.878645 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.878623    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"
Sep 10 20:37:04.878984 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.878961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 20:37:04.880319 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.880299    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-w5d4j"]
Sep 10 20:37:04.880660 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.880644    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:04.882373 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.882355    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 20:37:04.882498 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.882482    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4hjxk\""
Sep 10 20:37:04.884286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.884268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 20:37:04.884675 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.884660    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 20:37:04.884767 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.884753    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 20:37:04.886349 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.886331    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drws7\""
Sep 10 20:37:04.887177 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.887160    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:37:04.890785 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.890765    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 20:37:04.903450 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.903431    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"]
Sep 10 20:37:04.905164 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.905142    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55db58887f-8zf5r"]
Sep 10 20:37:04.933564 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.933502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxppv\" (UniqueName: \"kubernetes.io/projected/c0845dc6-da33-455e-9c95-4c9776458de3-kube-api-access-fxppv\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.933564 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.933534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0845dc6-da33-455e-9c95-4c9776458de3-config\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.933564 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.933550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0845dc6-da33-455e-9c95-4c9776458de3-serving-cert\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.933804 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.933593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0845dc6-da33-455e-9c95-4c9776458de3-trusted-ca\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.934227 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.934206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0845dc6-da33-455e-9c95-4c9776458de3-config\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.934302 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.934286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0845dc6-da33-455e-9c95-4c9776458de3-trusted-ca\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.936593 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.936575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0845dc6-da33-455e-9c95-4c9776458de3-serving-cert\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.956357 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.956335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxppv\" (UniqueName: \"kubernetes.io/projected/c0845dc6-da33-455e-9c95-4c9776458de3-kube-api-access-fxppv\") pod \"console-operator-696884fb9c-zz76z\" (UID: \"c0845dc6-da33-455e-9c95-4c9776458de3\") " pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:04.969292 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:04.969270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:05.034910 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.034885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7b5\" (UniqueName: \"kubernetes.io/projected/3992e74c-f2ea-432a-8b36-bdfe17d9766a-kube-api-access-ch7b5\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.034997 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.034928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-registry-certificates\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.034997 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.034957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-trusted-ca\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.034997 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.034976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-installation-pull-secrets\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.034997 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.034990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3992e74c-f2ea-432a-8b36-bdfe17d9766a-snapshots\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.035148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-bound-sa-token\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.035148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9pjh\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-kube-api-access-w9pjh\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.035148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3992e74c-f2ea-432a-8b36-bdfe17d9766a-tmp\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.035308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3992e74c-f2ea-432a-8b36-bdfe17d9766a-service-ca-bundle\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.035308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3992e74c-f2ea-432a-8b36-bdfe17d9766a-serving-cert\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.035308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035194    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-image-registry-private-configuration\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.035308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3992e74c-f2ea-432a-8b36-bdfe17d9766a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.035308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nqt\" (UniqueName: \"kubernetes.io/projected/0bc0a8b6-f59d-43b0-b028-d5c604f079f1-kube-api-access-77nqt\") pod \"volume-data-source-validator-86695b7cc7-sdfnk\" (UID: \"0bc0a8b6-f59d-43b0-b028-d5c604f079f1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"
Sep 10 20:37:05.035308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.035600 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.035327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f23031-8013-490d-b07b-a215fe3ed40b-ca-trust-extracted\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.136317 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-image-registry-private-configuration\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3992e74c-f2ea-432a-8b36-bdfe17d9766a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nqt\" (UniqueName: \"kubernetes.io/projected/0bc0a8b6-f59d-43b0-b028-d5c604f079f1-kube-api-access-77nqt\") pod \"volume-data-source-validator-86695b7cc7-sdfnk\" (UID: \"0bc0a8b6-f59d-43b0-b028-d5c604f079f1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f23031-8013-490d-b07b-a215fe3ed40b-ca-trust-extracted\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7b5\" (UniqueName: \"kubernetes.io/projected/3992e74c-f2ea-432a-8b36-bdfe17d9766a-kube-api-access-ch7b5\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-registry-certificates\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:05.136516    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:05.136537    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55db58887f-8zf5r: secret "image-registry-tls" not found
Sep 10 20:37:05.136615 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:05.136611    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls podName:13f23031-8013-490d-b07b-a215fe3ed40b nodeName:}" failed. No retries permitted until 2026-09-10 20:37:05.636589676 +0000 UTC m=+61.547626957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls") pod "image-registry-55db58887f-8zf5r" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b") : secret "image-registry-tls" not found
Sep 10 20:37:05.137090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-trusted-ca\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.137090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-installation-pull-secrets\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.137090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3992e74c-f2ea-432a-8b36-bdfe17d9766a-snapshots\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.137090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-bound-sa-token\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.137090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9pjh\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-kube-api-access-w9pjh\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.137090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f23031-8013-490d-b07b-a215fe3ed40b-ca-trust-extracted\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.137090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.136896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3992e74c-f2ea-432a-8b36-bdfe17d9766a-tmp\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.137536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-registry-certificates\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.137536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3992e74c-f2ea-432a-8b36-bdfe17d9766a-service-ca-bundle\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.137536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3992e74c-f2ea-432a-8b36-bdfe17d9766a-tmp\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.137536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3992e74c-f2ea-432a-8b36-bdfe17d9766a-serving-cert\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.137536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3992e74c-f2ea-432a-8b36-bdfe17d9766a-snapshots\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.137536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-trusted-ca\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.137536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137536    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3992e74c-f2ea-432a-8b36-bdfe17d9766a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.137844 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.137792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3992e74c-f2ea-432a-8b36-bdfe17d9766a-service-ca-bundle\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.138878 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.138861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-image-registry-private-configuration\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.139347 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.139328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-installation-pull-secrets\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.139698 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.139670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3992e74c-f2ea-432a-8b36-bdfe17d9766a-serving-cert\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.153167 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.153147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9pjh\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-kube-api-access-w9pjh\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.154213 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.154190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-bound-sa-token\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.154359 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.154344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7b5\" (UniqueName: \"kubernetes.io/projected/3992e74c-f2ea-432a-8b36-bdfe17d9766a-kube-api-access-ch7b5\") pod \"insights-operator-5844b856db-w5d4j\" (UID: \"3992e74c-f2ea-432a-8b36-bdfe17d9766a\") " pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.155206 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.155188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nqt\" (UniqueName: \"kubernetes.io/projected/0bc0a8b6-f59d-43b0-b028-d5c604f079f1-kube-api-access-77nqt\") pod \"volume-data-source-validator-86695b7cc7-sdfnk\" (UID: \"0bc0a8b6-f59d-43b0-b028-d5c604f079f1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"
Sep 10 20:37:05.174023 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.174004    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-w5d4j"
Sep 10 20:37:05.188807 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.188790    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"
Sep 10 20:37:05.252678 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.252655    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-zz76z"]
Sep 10 20:37:05.261048 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:05.261024    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0845dc6_da33_455e_9c95_4c9776458de3.slice/crio-65a17f0c6ac64ce5da83e8d04c454c83a1c303c693b0db60138877fd75b4e409 WatchSource:0}: Error finding container 65a17f0c6ac64ce5da83e8d04c454c83a1c303c693b0db60138877fd75b4e409: Status 404 returned error can't find the container with id 65a17f0c6ac64ce5da83e8d04c454c83a1c303c693b0db60138877fd75b4e409
Sep 10 20:37:05.446064 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.446019    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-w5d4j"]
Sep 10 20:37:05.447859 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:05.447835    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3992e74c_f2ea_432a_8b36_bdfe17d9766a.slice/crio-6d9429182a014f8c75ccce37d837ce6c4fd8ac5220295f32f3241c46a4db6d32 WatchSource:0}: Error finding container 6d9429182a014f8c75ccce37d837ce6c4fd8ac5220295f32f3241c46a4db6d32: Status 404 returned error can't find the container with id 6d9429182a014f8c75ccce37d837ce6c4fd8ac5220295f32f3241c46a4db6d32
Sep 10 20:37:05.465534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.465514    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk"]
Sep 10 20:37:05.467289 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:05.467265    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc0a8b6_f59d_43b0_b028_d5c604f079f1.slice/crio-45406c2281f9bf9dc10757bc07a8c8819359df1a150c86b035c095758a04ee79 WatchSource:0}: Error finding container 45406c2281f9bf9dc10757bc07a8c8819359df1a150c86b035c095758a04ee79: Status 404 returned error can't find the container with id 45406c2281f9bf9dc10757bc07a8c8819359df1a150c86b035c095758a04ee79
Sep 10 20:37:05.641917 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.641887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:05.642035 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:05.642021    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:37:05.642076 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:05.642039    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55db58887f-8zf5r: secret "image-registry-tls" not found
Sep 10 20:37:05.642116 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:05.642110    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls podName:13f23031-8013-490d-b07b-a215fe3ed40b nodeName:}" failed. No retries permitted until 2026-09-10 20:37:06.642095464 +0000 UTC m=+62.553132727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls") pod "image-registry-55db58887f-8zf5r" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b") : secret "image-registry-tls" not found
Sep 10 20:37:05.927680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.927641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-w5d4j" event={"ID":"3992e74c-f2ea-432a-8b36-bdfe17d9766a","Type":"ContainerStarted","Data":"6d9429182a014f8c75ccce37d837ce6c4fd8ac5220295f32f3241c46a4db6d32"}
Sep 10 20:37:05.929000 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.928974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk" event={"ID":"0bc0a8b6-f59d-43b0-b028-d5c604f079f1","Type":"ContainerStarted","Data":"45406c2281f9bf9dc10757bc07a8c8819359df1a150c86b035c095758a04ee79"}
Sep 10 20:37:05.930273 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:05.930249    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" event={"ID":"c0845dc6-da33-455e-9c95-4c9776458de3","Type":"ContainerStarted","Data":"65a17f0c6ac64ce5da83e8d04c454c83a1c303c693b0db60138877fd75b4e409"}
Sep 10 20:37:06.653291 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:06.653249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:06.653467 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:06.653425    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:37:06.653467 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:06.653444    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55db58887f-8zf5r: secret "image-registry-tls" not found
Sep 10 20:37:06.653576 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:06.653512    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls podName:13f23031-8013-490d-b07b-a215fe3ed40b nodeName:}" failed. No retries permitted until 2026-09-10 20:37:08.65349129 +0000 UTC m=+64.564528572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls") pod "image-registry-55db58887f-8zf5r" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b") : secret "image-registry-tls" not found
Sep 10 20:37:08.667006 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.666975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:08.667284 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:08.667104    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:37:08.667284 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:08.667123    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55db58887f-8zf5r: secret "image-registry-tls" not found
Sep 10 20:37:08.667284 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:08.667172    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls podName:13f23031-8013-490d-b07b-a215fe3ed40b nodeName:}" failed. No retries permitted until 2026-09-10 20:37:12.667156451 +0000 UTC m=+68.578193713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls") pod "image-registry-55db58887f-8zf5r" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b") : secret "image-registry-tls" not found
Sep 10 20:37:08.746770 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.746733    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"]
Sep 10 20:37:08.761248 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.761210    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"]
Sep 10 20:37:08.761360 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.761302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"
Sep 10 20:37:08.764391 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.764369    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5755v\""
Sep 10 20:37:08.764482 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.764372    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 20:37:08.766083 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.766066    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 20:37:08.767311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.767288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrvf\" (UniqueName: \"kubernetes.io/projected/e37f13fb-6890-40de-8fff-043d0e42a7df-kube-api-access-9xrvf\") pod \"migrator-8b4d697c5-rtbq7\" (UID: \"e37f13fb-6890-40de-8fff-043d0e42a7df\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"
Sep 10 20:37:08.868329 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.868305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrvf\" (UniqueName: \"kubernetes.io/projected/e37f13fb-6890-40de-8fff-043d0e42a7df-kube-api-access-9xrvf\") pod \"migrator-8b4d697c5-rtbq7\" (UID: \"e37f13fb-6890-40de-8fff-043d0e42a7df\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"
Sep 10 20:37:08.879822 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.879803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrvf\" (UniqueName: \"kubernetes.io/projected/e37f13fb-6890-40de-8fff-043d0e42a7df-kube-api-access-9xrvf\") pod \"migrator-8b4d697c5-rtbq7\" (UID: \"e37f13fb-6890-40de-8fff-043d0e42a7df\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"
Sep 10 20:37:08.940207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.940174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-w5d4j" event={"ID":"3992e74c-f2ea-432a-8b36-bdfe17d9766a","Type":"ContainerStarted","Data":"3481526ac0413601b7d61be4cd82c4995531d8f2e33f45126b0a021e330a812e"}
Sep 10 20:37:08.941654 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.941632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk" event={"ID":"0bc0a8b6-f59d-43b0-b028-d5c604f079f1","Type":"ContainerStarted","Data":"64d3f863c4254e0c16c52c7c3bdd72babf9d6507fc930efe7d512af5e1215d6b"}
Sep 10 20:37:08.943194 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.943177    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/0.log"
Sep 10 20:37:08.943258 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.943209    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0845dc6-da33-455e-9c95-4c9776458de3" containerID="745892dd4ca98e03c637bb90dc56b69371a8dd664911e1172e35dce5b5b0bdbe" exitCode=255
Sep 10 20:37:08.943258 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.943242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" event={"ID":"c0845dc6-da33-455e-9c95-4c9776458de3","Type":"ContainerDied","Data":"745892dd4ca98e03c637bb90dc56b69371a8dd664911e1172e35dce5b5b0bdbe"}
Sep 10 20:37:08.943411 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.943400    2565 scope.go:117] "RemoveContainer" containerID="745892dd4ca98e03c637bb90dc56b69371a8dd664911e1172e35dce5b5b0bdbe"
Sep 10 20:37:08.959785 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.959736    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-w5d4j" podStartSLOduration=1.853591327 podStartE2EDuration="4.959716967s" totalImagesPullingTime="3.10612564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:08.959451112 +0000 UTC m=+64.870488409" watchObservedRunningTime="2026-09-10 20:37:08.959716967 +0000 UTC m=+64.870754255"
Sep 10 20:37:08.978651 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:08.978605    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sdfnk" podStartSLOduration=1.896261509 podStartE2EDuration="4.978593033s" totalImagesPullingTime="3.082331524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:08.977992039 +0000 UTC m=+64.889029323" watchObservedRunningTime="2026-09-10 20:37:08.978593033 +0000 UTC m=+64.889630341"
Sep 10 20:37:09.092632 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.092607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"
Sep 10 20:37:09.270727 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.270669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:37:09.270854 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.270735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:37:09.270854 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.270822    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:37:09.270919 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.270877    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls podName:b492afb9-cddd-4ef5-a90e-6c89771312a9 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:41.27086269 +0000 UTC m=+97.181899952 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls") pod "dns-default-hkl9m" (UID: "b492afb9-cddd-4ef5-a90e-6c89771312a9") : secret "dns-default-metrics-tls" not found
Sep 10 20:37:09.274109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.274090    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:37:09.281925 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.281906    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 20:37:09.281978 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.281955    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs podName:9171816d-e76b-42c2-bb56-e341aaed3db4 nodeName:}" failed. No retries permitted until 2026-09-10 20:38:13.28194201 +0000 UTC m=+129.192979272 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs") pod "network-metrics-daemon-kjnmq" (UID: "9171816d-e76b-42c2-bb56-e341aaed3db4") : secret "metrics-daemon-secret" not found
Sep 10 20:37:09.371900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.371849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:37:09.371900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.371887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:37:09.378274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.378257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:37:09.378626 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.378613    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:37:09.384869 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.384850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/159d4776-28fe-4b94-8cc4-85a239b5a5ba-original-pull-secret\") pod \"global-pull-secret-syncer-twjd6\" (UID: \"159d4776-28fe-4b94-8cc4-85a239b5a5ba\") " pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:37:09.386159 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.386144    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:37:09.396086 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.396040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92bjl\" (UniqueName: \"kubernetes.io/projected/f705213e-23d4-458f-a7a8-ff4d4c70b21c-kube-api-access-92bjl\") pod \"network-check-target-ppwj5\" (UID: \"f705213e-23d4-458f-a7a8-ff4d4c70b21c\") " pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:37:09.397022 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.397000    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7"]
Sep 10 20:37:09.399229 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:09.399204    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37f13fb_6890_40de_8fff_043d0e42a7df.slice/crio-26b7666aca525dd3ca9b29589e08adb4bf49c69c1e0e1c61cd7a33f1d167a709 WatchSource:0}: Error finding container 26b7666aca525dd3ca9b29589e08adb4bf49c69c1e0e1c61cd7a33f1d167a709: Status 404 returned error can't find the container with id 26b7666aca525dd3ca9b29589e08adb4bf49c69c1e0e1c61cd7a33f1d167a709
Sep 10 20:37:09.408277 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.408259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twjd6"
Sep 10 20:37:09.702678 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.702373    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24xx\""
Sep 10 20:37:09.707211 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.706991    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:37:09.708941 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.708917    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-twjd6"]
Sep 10 20:37:09.715070 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:09.715041    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159d4776_28fe_4b94_8cc4_85a239b5a5ba.slice/crio-0f08fe71360aec96c7ce55326f208d0ba63357c52c99b6adbf36f73e2a5cdd27 WatchSource:0}: Error finding container 0f08fe71360aec96c7ce55326f208d0ba63357c52c99b6adbf36f73e2a5cdd27: Status 404 returned error can't find the container with id 0f08fe71360aec96c7ce55326f208d0ba63357c52c99b6adbf36f73e2a5cdd27
Sep 10 20:37:09.752645 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.752619    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8td6j"]
Sep 10 20:37:09.755349 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.755316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.758142 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.758117    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:37:09.758491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.758466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:37:09.758567 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.758515    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-djk6c\""
Sep 10 20:37:09.766598 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.766533    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8td6j"]
Sep 10 20:37:09.775979 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.775955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7227f57-b436-4625-bf3a-20086578fde4-crio-socket\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.776095 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.776016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.776095 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.776052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.776095 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.776079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2q9\" (UniqueName: \"kubernetes.io/projected/b7227f57-b436-4625-bf3a-20086578fde4-kube-api-access-ph2q9\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.776250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.776232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7227f57-b436-4625-bf3a-20086578fde4-data-volume\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.877418 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.877392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7227f57-b436-4625-bf3a-20086578fde4-crio-socket\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.877550 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.877443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.877550 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.877520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7227f57-b436-4625-bf3a-20086578fde4-crio-socket\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.877674 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.877554    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:09.877674 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.877565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.877674 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.877597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2q9\" (UniqueName: \"kubernetes.io/projected/b7227f57-b436-4625-bf3a-20086578fde4-kube-api-access-ph2q9\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.877674 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.877623    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:10.377603444 +0000 UTC m=+66.288640715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:09.877674 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.877657    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:09.877859 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.877720    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:10.377706235 +0000 UTC m=+66.288743500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:09.877859 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.877741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7227f57-b436-4625-bf3a-20086578fde4-data-volume\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.877986 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.877972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7227f57-b436-4625-bf3a-20086578fde4-data-volume\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.886572 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.886551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2q9\" (UniqueName: \"kubernetes.io/projected/b7227f57-b436-4625-bf3a-20086578fde4-kube-api-access-ph2q9\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:09.948365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.948324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-twjd6" event={"ID":"159d4776-28fe-4b94-8cc4-85a239b5a5ba","Type":"ContainerStarted","Data":"0f08fe71360aec96c7ce55326f208d0ba63357c52c99b6adbf36f73e2a5cdd27"}
Sep 10 20:37:09.950123 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.950102    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:37:09.950647 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.950593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/0.log"
Sep 10 20:37:09.950647 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.950634    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0845dc6-da33-455e-9c95-4c9776458de3" containerID="be8ca29e87609d573369289bb8e6484a0b6f99d51e172ec62017fa99a04f2cc9" exitCode=255
Sep 10 20:37:09.950820 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.950732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" event={"ID":"c0845dc6-da33-455e-9c95-4c9776458de3","Type":"ContainerDied","Data":"be8ca29e87609d573369289bb8e6484a0b6f99d51e172ec62017fa99a04f2cc9"}
Sep 10 20:37:09.950820 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.950780    2565 scope.go:117] "RemoveContainer" containerID="745892dd4ca98e03c637bb90dc56b69371a8dd664911e1172e35dce5b5b0bdbe"
Sep 10 20:37:09.951003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.950969    2565 scope.go:117] "RemoveContainer" containerID="be8ca29e87609d573369289bb8e6484a0b6f99d51e172ec62017fa99a04f2cc9"
Sep 10 20:37:09.951222 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:09.951176    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-zz76z_openshift-console-operator(c0845dc6-da33-455e-9c95-4c9776458de3)\"" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" podUID="c0845dc6-da33-455e-9c95-4c9776458de3"
Sep 10 20:37:09.952096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.952068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7" event={"ID":"e37f13fb-6890-40de-8fff-043d0e42a7df","Type":"ContainerStarted","Data":"26b7666aca525dd3ca9b29589e08adb4bf49c69c1e0e1c61cd7a33f1d167a709"}
Sep 10 20:37:09.986246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:09.986225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ppwj5"]
Sep 10 20:37:09.990632 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:09.990609    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf705213e_23d4_458f_a7a8_ff4d4c70b21c.slice/crio-4f4ed5807939ea51b48da895f91728422cde745853aadde0ab3b93cc7b828011 WatchSource:0}: Error finding container 4f4ed5807939ea51b48da895f91728422cde745853aadde0ab3b93cc7b828011: Status 404 returned error can't find the container with id 4f4ed5807939ea51b48da895f91728422cde745853aadde0ab3b93cc7b828011
Sep 10 20:37:10.381933 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:10.381895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:10.382110 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:10.381941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:10.382110 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:10.382023    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:10.382110 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:10.382048    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:10.382110 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:10.382090    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:11.382070686 +0000 UTC m=+67.293107967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:10.382110 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:10.382110    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:11.382100983 +0000 UTC m=+67.293138246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:10.961293 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:10.961219    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:37:10.961674 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:10.961622    2565 scope.go:117] "RemoveContainer" containerID="be8ca29e87609d573369289bb8e6484a0b6f99d51e172ec62017fa99a04f2cc9"
Sep 10 20:37:10.961843 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:10.961813    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-zz76z_openshift-console-operator(c0845dc6-da33-455e-9c95-4c9776458de3)\"" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" podUID="c0845dc6-da33-455e-9c95-4c9776458de3"
Sep 10 20:37:10.964321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:10.964294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7" event={"ID":"e37f13fb-6890-40de-8fff-043d0e42a7df","Type":"ContainerStarted","Data":"5b657c14cc0a20c4948f40a35b917ec30c2c6ebd4fcc1378b675a85453f76f3c"}
Sep 10 20:37:10.964415 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:10.964352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7" event={"ID":"e37f13fb-6890-40de-8fff-043d0e42a7df","Type":"ContainerStarted","Data":"6ecba0299eeb9b299bba14a88a2036043f06441d2342cc7989d181d5fca18e1c"}
Sep 10 20:37:10.966154 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:10.966099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ppwj5" event={"ID":"f705213e-23d4-458f-a7a8-ff4d4c70b21c","Type":"ContainerStarted","Data":"4f4ed5807939ea51b48da895f91728422cde745853aadde0ab3b93cc7b828011"}
Sep 10 20:37:11.013310 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:11.013257    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rtbq7" podStartSLOduration=1.788511996 podStartE2EDuration="3.013239877s" totalImagesPullingTime="1.224727881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:11.012592537 +0000 UTC m=+66.923629822" watchObservedRunningTime="2026-09-10 20:37:11.013239877 +0000 UTC m=+66.924277163"
Sep 10 20:37:11.392578 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:11.392544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:11.392982 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:11.392586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:11.392982 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:11.392766    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:11.392982 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:11.392820    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:11.392982 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:11.392843    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:13.392825811 +0000 UTC m=+69.303863076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:11.392982 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:11.392873    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:13.392858067 +0000 UTC m=+69.303895333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:12.703714 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:12.703665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:12.704132 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:12.703798    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:37:12.704132 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:12.703813    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55db58887f-8zf5r: secret "image-registry-tls" not found
Sep 10 20:37:12.704132 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:12.703865    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls podName:13f23031-8013-490d-b07b-a215fe3ed40b nodeName:}" failed. No retries permitted until 2026-09-10 20:37:20.703847785 +0000 UTC m=+76.614885049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls") pod "image-registry-55db58887f-8zf5r" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b") : secret "image-registry-tls" not found
Sep 10 20:37:12.704132 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:12.704043    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vzjhd_5b91ad48-6bac-4b33-964b-f6cc1c5bada9/dns-node-resolver/0.log"
Sep 10 20:37:13.408996 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:13.408957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:13.408996 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:13.409000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:13.409253 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:13.409122    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:13.409253 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:13.409171    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:17.409158468 +0000 UTC m=+73.320195730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:13.409253 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:13.409122    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:13.409484 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:13.409464    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:17.409442515 +0000 UTC m=+73.320479795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:13.504113 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:13.504081    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-glc9l_c0d0e19d-c65f-4d91-9151-8340d44d2502/node-ca/0.log"
Sep 10 20:37:14.103781 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.103726    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bp5xz_dbfc039d-4acd-4649-a477-b18f2c799a86/serve-healthcheck-canary/0.log"
Sep 10 20:37:14.707242 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.707211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rtbq7_e37f13fb-6890-40de-8fff-043d0e42a7df/migrator/0.log"
Sep 10 20:37:14.903640 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.903614    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rtbq7_e37f13fb-6890-40de-8fff-043d0e42a7df/graceful-termination/0.log"
Sep 10 20:37:14.969813 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.969756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:14.969909 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.969823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:14.970123 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.970111    2565 scope.go:117] "RemoveContainer" containerID="be8ca29e87609d573369289bb8e6484a0b6f99d51e172ec62017fa99a04f2cc9"
Sep 10 20:37:14.970255 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:14.970239    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-zz76z_openshift-console-operator(c0845dc6-da33-455e-9c95-4c9776458de3)\"" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" podUID="c0845dc6-da33-455e-9c95-4c9776458de3"
Sep 10 20:37:14.981995 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.981967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ppwj5" event={"ID":"f705213e-23d4-458f-a7a8-ff4d4c70b21c","Type":"ContainerStarted","Data":"88c5d11094e37c62ae19863c5f315e9e10dab116d5bfda8f4c14017ad3bb0f04"}
Sep 10 20:37:14.982117 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.982048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:37:14.983367 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.983345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-twjd6" event={"ID":"159d4776-28fe-4b94-8cc4-85a239b5a5ba","Type":"ContainerStarted","Data":"242c6eaca517efa359173339801f430ea9804c85b61df8670752234f5eff43bc"}
Sep 10 20:37:14.999635 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:14.999593    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ppwj5" podStartSLOduration=66.730983435 podStartE2EDuration="1m10.999582646s" totalImagesPullingTime="4.268599211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:14.99936896 +0000 UTC m=+70.910406245" watchObservedRunningTime="2026-09-10 20:37:14.999582646 +0000 UTC m=+70.910619929"
Sep 10 20:37:15.018586 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:15.018550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-twjd6" podStartSLOduration=66.469755576 podStartE2EDuration="1m11.018539959s" totalImagesPullingTime="4.548784383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:15.017875559 +0000 UTC m=+70.928912866" watchObservedRunningTime="2026-09-10 20:37:15.018539959 +0000 UTC m=+70.929577239"
Sep 10 20:37:17.438677 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:17.438635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:17.438677 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:17.438677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:17.439284 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:17.438790    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:17.439284 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:17.438850    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:25.438836029 +0000 UTC m=+81.349873291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:17.439284 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:17.438883    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:17.439284 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:17.438935    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:25.43892104 +0000 UTC m=+81.349958302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:20.762398 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:20.762361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:20.762819 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:20.762508    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:37:20.762819 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:20.762522    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55db58887f-8zf5r: secret "image-registry-tls" not found
Sep 10 20:37:20.762819 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:20.762574    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls podName:13f23031-8013-490d-b07b-a215fe3ed40b nodeName:}" failed. No retries permitted until 2026-09-10 20:37:36.762558325 +0000 UTC m=+92.673595607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls") pod "image-registry-55db58887f-8zf5r" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b") : secret "image-registry-tls" not found
Sep 10 20:37:25.497259 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:25.497218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:25.497259 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:25.497257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:25.497758 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:25.497368    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:25.497758 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:25.497432    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls podName:b7227f57-b436-4625-bf3a-20086578fde4 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:41.497417229 +0000 UTC m=+97.408454495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8td6j" (UID: "b7227f57-b436-4625-bf3a-20086578fde4") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:25.497849 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:25.497792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7227f57-b436-4625-bf3a-20086578fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:28.584872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:28.584736    2565 scope.go:117] "RemoveContainer" containerID="be8ca29e87609d573369289bb8e6484a0b6f99d51e172ec62017fa99a04f2cc9"
Sep 10 20:37:29.030445 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:29.030420    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:37:29.030608 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:29.030493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" event={"ID":"c0845dc6-da33-455e-9c95-4c9776458de3","Type":"ContainerStarted","Data":"31fb2bb6c4d1153a71a2467e8c0825d33ba0cdff8322e36a181061f09b51328b"}
Sep 10 20:37:29.030816 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:29.030780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:29.288817 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:29.288742    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-zz76z"
Sep 10 20:37:29.319321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:29.319273    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-zz76z" podStartSLOduration=22.031131524 podStartE2EDuration="25.319258038s" totalImagesPullingTime="3.288126514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:29.058179749 +0000 UTC m=+84.969217032" watchObservedRunningTime="2026-09-10 20:37:29.319258038 +0000 UTC m=+85.230295322"
Sep 10 20:37:35.553821 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.553787    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"]
Sep 10 20:37:35.556794 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.556772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.563407 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.563382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 20:37:35.563407 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.563395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l5thd\""
Sep 10 20:37:35.563407 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.563382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 20:37:35.564175 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.564161    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 20:37:35.565066 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.565053    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 20:37:35.583541 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.583521    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"]
Sep 10 20:37:35.605507 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.605485    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55db58887f-8zf5r"]
Sep 10 20:37:35.605704 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:35.605668    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-55db58887f-8zf5r" podUID="13f23031-8013-490d-b07b-a215fe3ed40b"
Sep 10 20:37:35.606438 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.606403    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-w9l65"]
Sep 10 20:37:35.609171 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.609156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-w9l65"
Sep 10 20:37:35.612666 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.612648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 20:37:35.613162 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.613145    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 20:37:35.613593 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.613573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tjc58\""
Sep 10 20:37:35.625214 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.625195    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-w9l65"]
Sep 10 20:37:35.672237 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.672189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgsb\" (UniqueName: \"kubernetes.io/projected/55e31363-71ba-496d-a745-1be9c1ae5d4f-kube-api-access-rcgsb\") pod \"managed-serviceaccount-addon-agent-568446c75d-98hq2\" (UID: \"55e31363-71ba-496d-a745-1be9c1ae5d4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.672358 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.672261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9hk\" (UniqueName: \"kubernetes.io/projected/4caa1c8e-a1de-4a18-8314-bda6d72b8147-kube-api-access-cb9hk\") pod \"downloads-76f99584f4-w9l65\" (UID: \"4caa1c8e-a1de-4a18-8314-bda6d72b8147\") " pod="openshift-console/downloads-76f99584f4-w9l65"
Sep 10 20:37:35.672426 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.672400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55e31363-71ba-496d-a745-1be9c1ae5d4f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-568446c75d-98hq2\" (UID: \"55e31363-71ba-496d-a745-1be9c1ae5d4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.773213 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.773183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgsb\" (UniqueName: \"kubernetes.io/projected/55e31363-71ba-496d-a745-1be9c1ae5d4f-kube-api-access-rcgsb\") pod \"managed-serviceaccount-addon-agent-568446c75d-98hq2\" (UID: \"55e31363-71ba-496d-a745-1be9c1ae5d4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.773401 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.773233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9hk\" (UniqueName: \"kubernetes.io/projected/4caa1c8e-a1de-4a18-8314-bda6d72b8147-kube-api-access-cb9hk\") pod \"downloads-76f99584f4-w9l65\" (UID: \"4caa1c8e-a1de-4a18-8314-bda6d72b8147\") " pod="openshift-console/downloads-76f99584f4-w9l65"
Sep 10 20:37:35.773401 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.773315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55e31363-71ba-496d-a745-1be9c1ae5d4f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-568446c75d-98hq2\" (UID: \"55e31363-71ba-496d-a745-1be9c1ae5d4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.775779 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.775759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55e31363-71ba-496d-a745-1be9c1ae5d4f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-568446c75d-98hq2\" (UID: \"55e31363-71ba-496d-a745-1be9c1ae5d4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.784559 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.784538    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgsb\" (UniqueName: \"kubernetes.io/projected/55e31363-71ba-496d-a745-1be9c1ae5d4f-kube-api-access-rcgsb\") pod \"managed-serviceaccount-addon-agent-568446c75d-98hq2\" (UID: \"55e31363-71ba-496d-a745-1be9c1ae5d4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.785002 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.784983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9hk\" (UniqueName: \"kubernetes.io/projected/4caa1c8e-a1de-4a18-8314-bda6d72b8147-kube-api-access-cb9hk\") pod \"downloads-76f99584f4-w9l65\" (UID: \"4caa1c8e-a1de-4a18-8314-bda6d72b8147\") " pod="openshift-console/downloads-76f99584f4-w9l65"
Sep 10 20:37:35.874879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.874854    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"
Sep 10 20:37:35.917937 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:35.917916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-w9l65"
Sep 10 20:37:36.050887 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.050862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:36.057603 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.057587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:36.159680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.159662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2"]
Sep 10 20:37:36.161408 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:36.161383    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55e31363_71ba_496d_a745_1be9c1ae5d4f.slice/crio-ed696173b88991ae62ea5bac0f2ac9edb1ed2dbb90bf5f7b45e8c9112a0744c8 WatchSource:0}: Error finding container ed696173b88991ae62ea5bac0f2ac9edb1ed2dbb90bf5f7b45e8c9112a0744c8: Status 404 returned error can't find the container with id ed696173b88991ae62ea5bac0f2ac9edb1ed2dbb90bf5f7b45e8c9112a0744c8
Sep 10 20:37:36.176063 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-bound-sa-token\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.176170 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f23031-8013-490d-b07b-a215fe3ed40b-ca-trust-extracted\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.176170 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176117    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9pjh\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-kube-api-access-w9pjh\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.176300 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-image-registry-private-configuration\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.176352 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176333    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-trusted-ca\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.176405 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-installation-pull-secrets\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.176405 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176392    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f23031-8013-490d-b07b-a215fe3ed40b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:37:36.176506 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-registry-certificates\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.176671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176648    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f23031-8013-490d-b07b-a215fe3ed40b-ca-trust-extracted\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:36.176813 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:37:36.176974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.176950    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:37:36.178639 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.178611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-kube-api-access-w9pjh" (OuterVolumeSpecName: "kube-api-access-w9pjh") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "kube-api-access-w9pjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:37:36.178639 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.178625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:37:36.178790 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.178734    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:37:36.178790 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.178773    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:37:36.195681 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.195660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-w9l65"]
Sep 10 20:37:36.198615 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:36.198595    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caa1c8e_a1de_4a18_8314_bda6d72b8147.slice/crio-87d55da812057c040bfb9c2b1cbc4e7ed09359c67a1f2a7a5f58e24c643417d3 WatchSource:0}: Error finding container 87d55da812057c040bfb9c2b1cbc4e7ed09359c67a1f2a7a5f58e24c643417d3: Status 404 returned error can't find the container with id 87d55da812057c040bfb9c2b1cbc4e7ed09359c67a1f2a7a5f58e24c643417d3
Sep 10 20:37:36.277584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.277554    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-installation-pull-secrets\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:36.277584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.277583    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-registry-certificates\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:36.277584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.277592    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-bound-sa-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:36.277766 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.277602    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9pjh\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-kube-api-access-w9pjh\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:36.277766 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.277611    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f23031-8013-490d-b07b-a215fe3ed40b-image-registry-private-configuration\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:36.277766 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.277619    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f23031-8013-490d-b07b-a215fe3ed40b-trusted-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:36.782116 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.782064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:36.785613 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.785589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"image-registry-55db58887f-8zf5r\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") " pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:36.882779 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.882714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") pod \"13f23031-8013-490d-b07b-a215fe3ed40b\" (UID: \"13f23031-8013-490d-b07b-a215fe3ed40b\") "
Sep 10 20:37:36.884830 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.884803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13f23031-8013-490d-b07b-a215fe3ed40b" (UID: "13f23031-8013-490d-b07b-a215fe3ed40b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:37:36.983472 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:36.983448    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f23031-8013-490d-b07b-a215fe3ed40b-registry-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:37:37.059519 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:37.059482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-w9l65" event={"ID":"4caa1c8e-a1de-4a18-8314-bda6d72b8147","Type":"ContainerStarted","Data":"87d55da812057c040bfb9c2b1cbc4e7ed09359c67a1f2a7a5f58e24c643417d3"}
Sep 10 20:37:37.061232 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:37.061204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2" event={"ID":"55e31363-71ba-496d-a745-1be9c1ae5d4f","Type":"ContainerStarted","Data":"ed696173b88991ae62ea5bac0f2ac9edb1ed2dbb90bf5f7b45e8c9112a0744c8"}
Sep 10 20:37:37.061338 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:37.061252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55db58887f-8zf5r"
Sep 10 20:37:37.106234 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:37.106190    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55db58887f-8zf5r"]
Sep 10 20:37:37.109964 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:37.109938    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55db58887f-8zf5r"]
Sep 10 20:37:38.590152 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:38.590116    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f23031-8013-490d-b07b-a215fe3ed40b" path="/var/lib/kubelet/pods/13f23031-8013-490d-b07b-a215fe3ed40b/volumes"
Sep 10 20:37:40.074192 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.074152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2" event={"ID":"55e31363-71ba-496d-a745-1be9c1ae5d4f","Type":"ContainerStarted","Data":"fa8e47499a937250596448ce725c89ec9162cdaea9080dffe430cb3b5129226e"}
Sep 10 20:37:40.097524 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.097472    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-568446c75d-98hq2" podStartSLOduration=2.153076902 podStartE2EDuration="5.09745592s" totalImagesPullingTime="2.944379018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:40.096633594 +0000 UTC m=+96.007670879" watchObservedRunningTime="2026-09-10 20:37:40.09745592 +0000 UTC m=+96.008493204"
Sep 10 20:37:40.771594 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.771562    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"]
Sep 10 20:37:40.774718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.774702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:40.777696 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.777658    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 20:37:40.777804 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.777669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hl5nx\""
Sep 10 20:37:40.787245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.787225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"]
Sep 10 20:37:40.914421 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:40.914389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b90fe30-6689-4dc5-a19c-b528aa6fdce1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-m6lv6\" (UID: \"4b90fe30-6689-4dc5-a19c-b528aa6fdce1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:41.015047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.015015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b90fe30-6689-4dc5-a19c-b528aa6fdce1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-m6lv6\" (UID: \"4b90fe30-6689-4dc5-a19c-b528aa6fdce1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:41.015203 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:41.015181    2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 20:37:41.015313 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:37:41.015300    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b90fe30-6689-4dc5-a19c-b528aa6fdce1-tls-certificates podName:4b90fe30-6689-4dc5-a19c-b528aa6fdce1 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:41.515278875 +0000 UTC m=+97.426316140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4b90fe30-6689-4dc5-a19c-b528aa6fdce1-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-m6lv6" (UID: "4b90fe30-6689-4dc5-a19c-b528aa6fdce1") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 20:37:41.317640 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.317610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:37:41.320075 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.320051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b492afb9-cddd-4ef5-a90e-6c89771312a9-metrics-tls\") pod \"dns-default-hkl9m\" (UID: \"b492afb9-cddd-4ef5-a90e-6c89771312a9\") " pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:37:41.377286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.377262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-89pjq\""
Sep 10 20:37:41.384431 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.384409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:37:41.519610 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.519575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b90fe30-6689-4dc5-a19c-b528aa6fdce1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-m6lv6\" (UID: \"4b90fe30-6689-4dc5-a19c-b528aa6fdce1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:41.519767 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.519627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:41.522207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.522183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b90fe30-6689-4dc5-a19c-b528aa6fdce1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-m6lv6\" (UID: \"4b90fe30-6689-4dc5-a19c-b528aa6fdce1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:41.522329 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.522229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7227f57-b436-4625-bf3a-20086578fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8td6j\" (UID: \"b7227f57-b436-4625-bf3a-20086578fde4\") " pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:41.570155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.570096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8td6j"
Sep 10 20:37:41.684591 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.684547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:41.684767 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.684734    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hkl9m"]
Sep 10 20:37:41.687516 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:41.687491    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb492afb9_cddd_4ef5_a90e_6c89771312a9.slice/crio-bb5f6732831b1c767847cdeb4d594a6177158ec9603fe22ed1e7c4de6fb20924 WatchSource:0}: Error finding container bb5f6732831b1c767847cdeb4d594a6177158ec9603fe22ed1e7c4de6fb20924: Status 404 returned error can't find the container with id bb5f6732831b1c767847cdeb4d594a6177158ec9603fe22ed1e7c4de6fb20924
Sep 10 20:37:41.868579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.868547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8td6j"]
Sep 10 20:37:41.871604 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:41.871580    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7227f57_b436_4625_bf3a_20086578fde4.slice/crio-cd485228548b2b3c5443c7dd642a0b90f46c661622d86fa2c9422f6da5af045f WatchSource:0}: Error finding container cd485228548b2b3c5443c7dd642a0b90f46c661622d86fa2c9422f6da5af045f: Status 404 returned error can't find the container with id cd485228548b2b3c5443c7dd642a0b90f46c661622d86fa2c9422f6da5af045f
Sep 10 20:37:41.985134 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:41.985107    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"]
Sep 10 20:37:41.987700 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:41.987660    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b90fe30_6689_4dc5_a19c_b528aa6fdce1.slice/crio-12c1b1c07da1404d4e2b1aa42a159660d0bf9742975b138239d1a273fa98235b WatchSource:0}: Error finding container 12c1b1c07da1404d4e2b1aa42a159660d0bf9742975b138239d1a273fa98235b: Status 404 returned error can't find the container with id 12c1b1c07da1404d4e2b1aa42a159660d0bf9742975b138239d1a273fa98235b
Sep 10 20:37:42.086511 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.086474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8td6j" event={"ID":"b7227f57-b436-4625-bf3a-20086578fde4","Type":"ContainerStarted","Data":"3ad47ac862d2efcc3fd0ce5949f371bd39a35116c3137a955b76871fafbf81cc"}
Sep 10 20:37:42.086652 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.086579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8td6j" event={"ID":"b7227f57-b436-4625-bf3a-20086578fde4","Type":"ContainerStarted","Data":"cd485228548b2b3c5443c7dd642a0b90f46c661622d86fa2c9422f6da5af045f"}
Sep 10 20:37:42.087817 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.087791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6" event={"ID":"4b90fe30-6689-4dc5-a19c-b528aa6fdce1","Type":"ContainerStarted","Data":"12c1b1c07da1404d4e2b1aa42a159660d0bf9742975b138239d1a273fa98235b"}
Sep 10 20:37:42.089150 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.089115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hkl9m" event={"ID":"b492afb9-cddd-4ef5-a90e-6c89771312a9","Type":"ContainerStarted","Data":"bb5f6732831b1c767847cdeb4d594a6177158ec9603fe22ed1e7c4de6fb20924"}
Sep 10 20:37:42.652896 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.652866    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8f65f69d5-qtxfc"]
Sep 10 20:37:42.682245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.682216    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f65f69d5-qtxfc"]
Sep 10 20:37:42.682413 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.682330    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.685530 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.685486    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 20:37:42.685669 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.685556    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 20:37:42.685669 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.685567    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 20:37:42.685802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.685743    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 20:37:42.685867 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.685826    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lrjhl\""
Sep 10 20:37:42.687106 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.687074    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 20:37:42.831417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.831016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-oauth-serving-cert\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.831417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.831074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-serving-cert\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.831417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.831152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-config\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.831417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.831176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-service-ca\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.831417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.831211    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-oauth-config\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.831417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.831251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhl28\" (UniqueName: \"kubernetes.io/projected/6afa84a1-0bbc-487b-81ed-3e9405e8b245-kube-api-access-zhl28\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.931800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-oauth-serving-cert\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.931839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-serving-cert\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.931889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-config\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.931911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-service-ca\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.931947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-oauth-config\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.931983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhl28\" (UniqueName: \"kubernetes.io/projected/6afa84a1-0bbc-487b-81ed-3e9405e8b245-kube-api-access-zhl28\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932701 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.932652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-oauth-serving-cert\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.932849 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.932823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-config\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.933089 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.933040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-service-ca\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.935458 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.935433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-serving-cert\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.936500 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.936457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-oauth-config\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.943453 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.943411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhl28\" (UniqueName: \"kubernetes.io/projected/6afa84a1-0bbc-487b-81ed-3e9405e8b245-kube-api-access-zhl28\") pod \"console-8f65f69d5-qtxfc\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") " pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:42.995368 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:42.995338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:37:44.789724 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:44.789668    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f65f69d5-qtxfc"]
Sep 10 20:37:44.792386 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:44.792359    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afa84a1_0bbc_487b_81ed_3e9405e8b245.slice/crio-51858bed67b6739ddc581217caca74396ceb3de3cfa2fca30473263b220bb321 WatchSource:0}: Error finding container 51858bed67b6739ddc581217caca74396ceb3de3cfa2fca30473263b220bb321: Status 404 returned error can't find the container with id 51858bed67b6739ddc581217caca74396ceb3de3cfa2fca30473263b220bb321
Sep 10 20:37:45.102804 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.102771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8td6j" event={"ID":"b7227f57-b436-4625-bf3a-20086578fde4","Type":"ContainerStarted","Data":"61e32dfbee6f37abb6ac05355e24693bd078f4e5326a6157c1280f7b8a97ccfc"}
Sep 10 20:37:45.104255 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.104228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f65f69d5-qtxfc" event={"ID":"6afa84a1-0bbc-487b-81ed-3e9405e8b245","Type":"ContainerStarted","Data":"51858bed67b6739ddc581217caca74396ceb3de3cfa2fca30473263b220bb321"}
Sep 10 20:37:45.106171 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.106146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6" event={"ID":"4b90fe30-6689-4dc5-a19c-b528aa6fdce1","Type":"ContainerStarted","Data":"d60575e7ba0892054e59a7495bf9ff7a32ee74015a109697447897790d89b3e6"}
Sep 10 20:37:45.106347 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.106326    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:45.108366 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.108299    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hkl9m" event={"ID":"b492afb9-cddd-4ef5-a90e-6c89771312a9","Type":"ContainerStarted","Data":"f08c5896056f9f7832836b035020e2ab8ec9555431ecae661f8b19febe77f712"}
Sep 10 20:37:45.108366 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.108341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hkl9m" event={"ID":"b492afb9-cddd-4ef5-a90e-6c89771312a9","Type":"ContainerStarted","Data":"6420c61553be58ce809b6f996defa7a59a62df5766a9b73e56fd4637d6af714d"}
Sep 10 20:37:45.108513 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.108446    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:37:45.111581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.111559    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6"
Sep 10 20:37:45.131800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.131711    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-m6lv6" podStartSLOduration=2.601820419 podStartE2EDuration="5.131697646s" totalImagesPullingTime="2.529877227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:45.130883518 +0000 UTC m=+101.041920814" watchObservedRunningTime="2026-09-10 20:37:45.131697646 +0000 UTC m=+101.042734923"
Sep 10 20:37:45.175349 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.175302    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hkl9m" podStartSLOduration=65.346403218 podStartE2EDuration="1m8.175284072s" totalImagesPullingTime="2.828880854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:45.175053009 +0000 UTC m=+101.086090294" watchObservedRunningTime="2026-09-10 20:37:45.175284072 +0000 UTC m=+101.086321358"
Sep 10 20:37:45.865358 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.865192    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"]
Sep 10 20:37:45.887082 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.887047    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"]
Sep 10 20:37:45.887257 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.887159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:45.891409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.891233    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 20:37:45.895049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.893485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8gj8z\""
Sep 10 20:37:45.895049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.893699    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 20:37:45.895049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.893900    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 20:37:45.895049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.894107    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 20:37:45.895049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.894302    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 20:37:45.957570 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.957537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pqpz\" (UniqueName: \"kubernetes.io/projected/0e35d50f-376e-4777-a7f0-7aa1d88948b5-kube-api-access-5pqpz\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:45.957745 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.957585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e35d50f-376e-4777-a7f0-7aa1d88948b5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:45.957745 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.957636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35d50f-376e-4777-a7f0-7aa1d88948b5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:45.957745 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.957667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35d50f-376e-4777-a7f0-7aa1d88948b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:45.988845 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:45.988822    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ppwj5"
Sep 10 20:37:46.058588 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.058552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pqpz\" (UniqueName: \"kubernetes.io/projected/0e35d50f-376e-4777-a7f0-7aa1d88948b5-kube-api-access-5pqpz\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.058754 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.058618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e35d50f-376e-4777-a7f0-7aa1d88948b5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.058754 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.058700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35d50f-376e-4777-a7f0-7aa1d88948b5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.058884 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.058736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35d50f-376e-4777-a7f0-7aa1d88948b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.060327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.060302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35d50f-376e-4777-a7f0-7aa1d88948b5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.063002 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.062940    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35d50f-376e-4777-a7f0-7aa1d88948b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.063189 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.063162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e35d50f-376e-4777-a7f0-7aa1d88948b5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.082738 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.082715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pqpz\" (UniqueName: \"kubernetes.io/projected/0e35d50f-376e-4777-a7f0-7aa1d88948b5-kube-api-access-5pqpz\") pod \"prometheus-operator-68bcd7f987-h5fl7\" (UID: \"0e35d50f-376e-4777-a7f0-7aa1d88948b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:46.204403 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:46.204328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"
Sep 10 20:37:54.596415 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:54.596385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7"]
Sep 10 20:37:54.598588 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:54.598560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e35d50f_376e_4777_a7f0_7aa1d88948b5.slice/crio-2b055b1b9bc05e8e8b0d2d86f28feb50115689ba03b3d739d7921e865b439683 WatchSource:0}: Error finding container 2b055b1b9bc05e8e8b0d2d86f28feb50115689ba03b3d739d7921e865b439683: Status 404 returned error can't find the container with id 2b055b1b9bc05e8e8b0d2d86f28feb50115689ba03b3d739d7921e865b439683
Sep 10 20:37:55.115751 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.115719    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hkl9m"
Sep 10 20:37:55.150030 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.149992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8td6j" event={"ID":"b7227f57-b436-4625-bf3a-20086578fde4","Type":"ContainerStarted","Data":"632890054c7b22d949499eee65a85961faa1de15f6686c75f467556b348fb4d2"}
Sep 10 20:37:55.152105 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.152070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f65f69d5-qtxfc" event={"ID":"6afa84a1-0bbc-487b-81ed-3e9405e8b245","Type":"ContainerStarted","Data":"de88ed6784fe99b4b5b7348c665c4632dfa986f71d13984857aa08b7bd60ba0e"}
Sep 10 20:37:55.154665 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.154633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-w9l65" event={"ID":"4caa1c8e-a1de-4a18-8314-bda6d72b8147","Type":"ContainerStarted","Data":"b2d6b4866ac3902083ef95a3e0ded585fb70e2e842dbbc07d4037069b6965681"}
Sep 10 20:37:55.155085 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.155062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-w9l65"
Sep 10 20:37:55.156669 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.156641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7" event={"ID":"0e35d50f-376e-4777-a7f0-7aa1d88948b5","Type":"ContainerStarted","Data":"2b055b1b9bc05e8e8b0d2d86f28feb50115689ba03b3d739d7921e865b439683"}
Sep 10 20:37:55.167608 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.167585    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-w9l65"
Sep 10 20:37:55.173833 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.173788    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8td6j" podStartSLOduration=34.047281507 podStartE2EDuration="46.173767296s" totalImagesPullingTime="12.126485789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:37:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:55.172816216 +0000 UTC m=+111.083853511" watchObservedRunningTime="2026-09-10 20:37:55.173767296 +0000 UTC m=+111.084804580"
Sep 10 20:37:55.194827 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.194784    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8f65f69d5-qtxfc" podStartSLOduration=3.653981788 podStartE2EDuration="13.194767614s" totalImagesPullingTime="9.540785826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:55.193726598 +0000 UTC m=+111.104763885" watchObservedRunningTime="2026-09-10 20:37:55.194767614 +0000 UTC m=+111.105804940"
Sep 10 20:37:55.214623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:55.214576    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-w9l65" podStartSLOduration=2.028399395 podStartE2EDuration="20.214562047s" totalImagesPullingTime="18.186162652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:55.213754623 +0000 UTC m=+111.124791908" watchObservedRunningTime="2026-09-10 20:37:55.214562047 +0000 UTC m=+111.125599330"
Sep 10 20:37:56.007559 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:56.007515    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f65f69d5-qtxfc"]
Sep 10 20:37:57.168560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:57.168515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7" event={"ID":"0e35d50f-376e-4777-a7f0-7aa1d88948b5","Type":"ContainerStarted","Data":"c09a2ccc12ce946d1c4c4e78e5d3ad3e6c762a110d8150b509d4896c9f1e8f88"}
Sep 10 20:37:57.169040 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:57.168587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7" event={"ID":"0e35d50f-376e-4777-a7f0-7aa1d88948b5","Type":"ContainerStarted","Data":"92fc2b83c3171bbbed1702ecdf062919e5bc8fb4967adc4145e07724845a5141"}
Sep 10 20:37:57.194958 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:57.194903    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-h5fl7" podStartSLOduration=10.319320348 podStartE2EDuration="12.194886688s" totalImagesPullingTime="1.87556634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:57.193506024 +0000 UTC m=+113.104543309" watchObservedRunningTime="2026-09-10 20:37:57.194886688 +0000 UTC m=+113.105923964"
Sep 10 20:37:59.331938 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.331901    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"]
Sep 10 20:37:59.366251 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.366220    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"]
Sep 10 20:37:59.366404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.366321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.369860 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.369838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h9b67\""
Sep 10 20:37:59.371233 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.371211    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 20:37:59.371327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.371235    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 20:37:59.378379 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.378354    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lkgvr"]
Sep 10 20:37:59.397619 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.397600    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.401042 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.401022    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 20:37:59.401160 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.401126    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pfdlc\""
Sep 10 20:37:59.401353 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.401336    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 20:37:59.411524 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.411504    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 20:37:59.470593 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.470559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae173ba-8df5-4919-9647-811bab52d34f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.470772 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.470600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae173ba-8df5-4919-9647-811bab52d34f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.470772 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.470711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqd8\" (UniqueName: \"kubernetes.io/projected/eae173ba-8df5-4919-9647-811bab52d34f-kube-api-access-gvqd8\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.470772 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.470751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae173ba-8df5-4919-9647-811bab52d34f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.572135 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae173ba-8df5-4919-9647-811bab52d34f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.572310 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d3249-e695-467c-af8f-74189a1013a9-metrics-client-ca\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572310 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-wtmp\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572310 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-sys\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572310 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-root\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-textfile\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae173ba-8df5-4919-9647-811bab52d34f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.572534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae173ba-8df5-4919-9647-811bab52d34f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.572534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlmzk\" (UniqueName: \"kubernetes.io/projected/4c1d3249-e695-467c-af8f-74189a1013a9-kube-api-access-xlmzk\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572803 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-tls\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572803 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.572803 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqd8\" (UniqueName: \"kubernetes.io/projected/eae173ba-8df5-4919-9647-811bab52d34f-kube-api-access-gvqd8\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.573000 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.572974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae173ba-8df5-4919-9647-811bab52d34f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.575243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.575223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae173ba-8df5-4919-9647-811bab52d34f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.575409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.575388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae173ba-8df5-4919-9647-811bab52d34f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.583093 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.583040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqd8\" (UniqueName: \"kubernetes.io/projected/eae173ba-8df5-4919-9647-811bab52d34f-kube-api-access-gvqd8\") pod \"openshift-state-metrics-67dcf8c979-tpgmn\" (UID: \"eae173ba-8df5-4919-9647-811bab52d34f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.672976 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.672945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673092 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d3249-e695-467c-af8f-74189a1013a9-metrics-client-ca\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673092 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-wtmp\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673205 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-wtmp\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673255 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-sys\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673255 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-root\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673352 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-sys\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c1d3249-e695-467c-af8f-74189a1013a9-root\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673453 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-textfile\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673497 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673548 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlmzk\" (UniqueName: \"kubernetes.io/projected/4c1d3249-e695-467c-af8f-74189a1013a9-kube-api-access-xlmzk\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673548 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-tls\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673809 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-textfile\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.673898 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.673850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.674323 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.674296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d3249-e695-467c-af8f-74189a1013a9-metrics-client-ca\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.677286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.677265    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"
Sep 10 20:37:59.677546 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.677521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-tls\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.677647 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.677553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c1d3249-e695-467c-af8f-74189a1013a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.683395 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.683370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlmzk\" (UniqueName: \"kubernetes.io/projected/4c1d3249-e695-467c-af8f-74189a1013a9-kube-api-access-xlmzk\") pod \"node-exporter-lkgvr\" (UID: \"4c1d3249-e695-467c-af8f-74189a1013a9\") " pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.707797 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.707776    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lkgvr"
Sep 10 20:37:59.731107 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:59.731080    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1d3249_e695_467c_af8f_74189a1013a9.slice/crio-8958cdc2440790772bb118707060e7a0fcbd8d8e9da980ce37bfd26ec8f61b2c WatchSource:0}: Error finding container 8958cdc2440790772bb118707060e7a0fcbd8d8e9da980ce37bfd26ec8f61b2c: Status 404 returned error can't find the container with id 8958cdc2440790772bb118707060e7a0fcbd8d8e9da980ce37bfd26ec8f61b2c
Sep 10 20:37:59.988084 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:37:59.988055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn"]
Sep 10 20:37:59.990620 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:37:59.990589    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae173ba_8df5_4919_9647_811bab52d34f.slice/crio-c7d5c6f1336e8c93e2b0594a705d038145f7a300c23fcb15524f4b1b9a6fecb8 WatchSource:0}: Error finding container c7d5c6f1336e8c93e2b0594a705d038145f7a300c23fcb15524f4b1b9a6fecb8: Status 404 returned error can't find the container with id c7d5c6f1336e8c93e2b0594a705d038145f7a300c23fcb15524f4b1b9a6fecb8
Sep 10 20:38:00.183029 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.182993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn" event={"ID":"eae173ba-8df5-4919-9647-811bab52d34f","Type":"ContainerStarted","Data":"a43bea7943e1433dee8e99209ab5efdbdcbfb8bb43109bb39cd23bde64c8034d"}
Sep 10 20:38:00.183159 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.183109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn" event={"ID":"eae173ba-8df5-4919-9647-811bab52d34f","Type":"ContainerStarted","Data":"c7d5c6f1336e8c93e2b0594a705d038145f7a300c23fcb15524f4b1b9a6fecb8"}
Sep 10 20:38:00.184481 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.184453    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkgvr" event={"ID":"4c1d3249-e695-467c-af8f-74189a1013a9","Type":"ContainerStarted","Data":"8958cdc2440790772bb118707060e7a0fcbd8d8e9da980ce37bfd26ec8f61b2c"}
Sep 10 20:38:00.481950 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.481480    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:38:00.496815 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.496340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.502557 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.502529    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 20:38:00.502807 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.502788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 20:38:00.503130 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.503111    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 20:38:00.503319 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.503303    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 20:38:00.503496 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.503478    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 20:38:00.504091 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.503787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 20:38:00.504091 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.503787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 20:38:00.504091 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.503918    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 20:38:00.504252 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.504230    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 20:38:00.504576 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.504413    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9cmlr\""
Sep 10 20:38:00.523160 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.515937    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:38:00.684217 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684217 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xwb\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-kube-api-access-99xwb\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-config-out\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684621 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684621 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684621 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684621 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684837 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-config-volume\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684837 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-web-config\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.684837 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.684713    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.785922 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.785922 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xwb\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-kube-api-access-99xwb\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.785922 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-config-out\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.785922 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.785922 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785840    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.785922 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.785992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.786018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-config-volume\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.786039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-web-config\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.786061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.786130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.786484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.787496    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.789539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.789825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.790311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-config-volume\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.791169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.791802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.791365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-config-out\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.792495 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.791825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.792552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.792519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.792654 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.792620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.793067 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.793044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-web-config\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.793856 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.793815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.797071 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.797032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xwb\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-kube-api-access-99xwb\") pod \"alertmanager-main-0\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:00.810413 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:00.810378    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:38:01.192230 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.192187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn" event={"ID":"eae173ba-8df5-4919-9647-811bab52d34f","Type":"ContainerStarted","Data":"a72114dbf50f04229f06a3797bb37b49e07991938af82dfe69f205045ae747ad"}
Sep 10 20:38:01.322641 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.322613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:38:01.332519 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:38:01.332488    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3d1682_647e_4a10_b610_62ea4936ed89.slice/crio-75786f19353a3c7788db2ca355a40dce8037ae7050d80462ac18e25bd731c321 WatchSource:0}: Error finding container 75786f19353a3c7788db2ca355a40dce8037ae7050d80462ac18e25bd731c321: Status 404 returned error can't find the container with id 75786f19353a3c7788db2ca355a40dce8037ae7050d80462ac18e25bd731c321
Sep 10 20:38:01.446809 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.446765    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"]
Sep 10 20:38:01.466741 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.466710    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.471179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.470190    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d45hiv6panofo\""
Sep 10 20:38:01.471179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.470214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 20:38:01.471179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.470469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-blgt2\""
Sep 10 20:38:01.471179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.470565    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 20:38:01.471179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.470814    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 20:38:01.471179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.470984    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 20:38:01.471179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.471019    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 20:38:01.474943 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.474630    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"]
Sep 10 20:38:01.594302 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.594713 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.594713 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f12e0c-0398-41b8-8aea-7b5675fe570b-metrics-client-ca\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.594713 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-tls\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.594713 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-grpc-tls\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.594713 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z54w\" (UniqueName: \"kubernetes.io/projected/d5f12e0c-0398-41b8-8aea-7b5675fe570b-kube-api-access-6z54w\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.594967 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.594967 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.594807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.695985 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.695955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.696080 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.695996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.696080 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.696032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.696080 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.696055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.696245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.696099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f12e0c-0398-41b8-8aea-7b5675fe570b-metrics-client-ca\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.696245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.696160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-tls\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.696245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.696199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-grpc-tls\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.696396 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.696286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z54w\" (UniqueName: \"kubernetes.io/projected/d5f12e0c-0398-41b8-8aea-7b5675fe570b-kube-api-access-6z54w\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.697480 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.697410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f12e0c-0398-41b8-8aea-7b5675fe570b-metrics-client-ca\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.698777 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.698753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.699917 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.699063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.699917 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.699421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.700470 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.700413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-tls\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.700620 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.700590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-grpc-tls\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.700764 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.700716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f12e0c-0398-41b8-8aea-7b5675fe570b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.706890 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.706871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z54w\" (UniqueName: \"kubernetes.io/projected/d5f12e0c-0398-41b8-8aea-7b5675fe570b-kube-api-access-6z54w\") pod \"thanos-querier-6f746ccff4-z8sh8\" (UID: \"d5f12e0c-0398-41b8-8aea-7b5675fe570b\") " pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:01.782207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:01.782182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:02.088677 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.088507    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"]
Sep 10 20:38:02.091526 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:38:02.091499    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f12e0c_0398_41b8_8aea_7b5675fe570b.slice/crio-31353eab270edfebea7eaafcc0b12bc92ba31ff6ec08bef6c6cd6b4cec1e0b2c WatchSource:0}: Error finding container 31353eab270edfebea7eaafcc0b12bc92ba31ff6ec08bef6c6cd6b4cec1e0b2c: Status 404 returned error can't find the container with id 31353eab270edfebea7eaafcc0b12bc92ba31ff6ec08bef6c6cd6b4cec1e0b2c
Sep 10 20:38:02.199052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.198975    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn" event={"ID":"eae173ba-8df5-4919-9647-811bab52d34f","Type":"ContainerStarted","Data":"2dd1d2c316be9d8550c24adace7c6fc3a1955805151c436828f8f3f276863ce1"}
Sep 10 20:38:02.200417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.200386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerStarted","Data":"75786f19353a3c7788db2ca355a40dce8037ae7050d80462ac18e25bd731c321"}
Sep 10 20:38:02.202221 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.202196    2565 generic.go:356] "Generic (PLEG): container finished" podID="4c1d3249-e695-467c-af8f-74189a1013a9" containerID="869df3ccae179ea6267234ae31942abcc36529656b9cfbb41498593c02103b56" exitCode=0
Sep 10 20:38:02.202337 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.202278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkgvr" event={"ID":"4c1d3249-e695-467c-af8f-74189a1013a9","Type":"ContainerDied","Data":"869df3ccae179ea6267234ae31942abcc36529656b9cfbb41498593c02103b56"}
Sep 10 20:38:02.209703 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.205912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" event={"ID":"d5f12e0c-0398-41b8-8aea-7b5675fe570b","Type":"ContainerStarted","Data":"31353eab270edfebea7eaafcc0b12bc92ba31ff6ec08bef6c6cd6b4cec1e0b2c"}
Sep 10 20:38:02.231054 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.231011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tpgmn" podStartSLOduration=1.534726879 podStartE2EDuration="3.230997747s" totalImagesPullingTime="1.696270868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:02.230928713 +0000 UTC m=+118.141965999" watchObservedRunningTime="2026-09-10 20:38:02.230997747 +0000 UTC m=+118.142035032"
Sep 10 20:38:02.996018 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:02.995944    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:38:03.212816 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.212767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkgvr" event={"ID":"4c1d3249-e695-467c-af8f-74189a1013a9","Type":"ContainerStarted","Data":"8d0f2545930f2a17abc340b9ef267a5d2d29580479fef19b4858840d3ca02690"}
Sep 10 20:38:03.212984 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.212836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkgvr" event={"ID":"4c1d3249-e695-467c-af8f-74189a1013a9","Type":"ContainerStarted","Data":"09c411b31ef412f591132b4cd7f8285b11a3fa9a343b3027967bb61ba34c584d"}
Sep 10 20:38:03.244307 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.244247    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lkgvr" podStartSLOduration=2.240306821 podStartE2EDuration="4.244230974s" totalImagesPullingTime="1.90715378s" totalInitContainerRuntime="96.770373ms" isStatefulPod=true podCreationTimestamp="2026-09-10 20:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:03.242587921 +0000 UTC m=+119.153625208" watchObservedRunningTime="2026-09-10 20:38:03.244230974 +0000 UTC m=+119.155268252"
Sep 10 20:38:03.797536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.797469    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"]
Sep 10 20:38:03.817548 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.817527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"]
Sep 10 20:38:03.817669 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.817610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:03.820967 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.820936    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 20:38:03.821089 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.821030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l6pcc\""
Sep 10 20:38:03.822610 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.822558    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d1l4rj2p09255\""
Sep 10 20:38:03.822610 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.822573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 20:38:03.822870 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.822850    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 20:38:03.822963 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.822900    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 20:38:03.916938 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.916910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqccg\" (UniqueName: \"kubernetes.io/projected/65148aca-66c0-450c-bdce-5dc0fa70e51c-kube-api-access-fqccg\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:03.917057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.916942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65148aca-66c0-450c-bdce-5dc0fa70e51c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:03.917057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.916971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-client-ca-bundle\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:03.917057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.916999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-secret-metrics-server-tls\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:03.917057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.917052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/65148aca-66c0-450c-bdce-5dc0fa70e51c-audit-log\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:03.917220 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.917076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-secret-metrics-server-client-certs\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:03.917220 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:03.917115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/65148aca-66c0-450c-bdce-5dc0fa70e51c-metrics-server-audit-profiles\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.017910 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.017877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/65148aca-66c0-450c-bdce-5dc0fa70e51c-metrics-server-audit-profiles\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018324 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.017933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqccg\" (UniqueName: \"kubernetes.io/projected/65148aca-66c0-450c-bdce-5dc0fa70e51c-kube-api-access-fqccg\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018324 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.017964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65148aca-66c0-450c-bdce-5dc0fa70e51c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018324 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.018007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-client-ca-bundle\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018324 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.018052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-secret-metrics-server-tls\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018555 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.018313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/65148aca-66c0-450c-bdce-5dc0fa70e51c-audit-log\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018555 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.018368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-secret-metrics-server-client-certs\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018765 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.018742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/65148aca-66c0-450c-bdce-5dc0fa70e51c-audit-log\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.018971 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.018923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65148aca-66c0-450c-bdce-5dc0fa70e51c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.019835 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.019788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/65148aca-66c0-450c-bdce-5dc0fa70e51c-metrics-server-audit-profiles\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.020983 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.020937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-secret-metrics-server-tls\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.021080 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.021065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-client-ca-bundle\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.022510 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.022467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/65148aca-66c0-450c-bdce-5dc0fa70e51c-secret-metrics-server-client-certs\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.028606 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.028581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqccg\" (UniqueName: \"kubernetes.io/projected/65148aca-66c0-450c-bdce-5dc0fa70e51c-kube-api-access-fqccg\") pod \"metrics-server-74b7d9f5dc-d28pw\" (UID: \"65148aca-66c0-450c-bdce-5dc0fa70e51c\") " pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.127369 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.127343    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:04.218112 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.218080    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerID="7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8" exitCode=0
Sep 10 20:38:04.218215 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.218174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8"}
Sep 10 20:38:04.425652 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:04.425624    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"]
Sep 10 20:38:04.943622 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:38:04.943590    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65148aca_66c0_450c_bdce_5dc0fa70e51c.slice/crio-e069ed014a89f919bb9e4a63ad6d18cab35773bb3a1fef78c356e661f2711da3 WatchSource:0}: Error finding container e069ed014a89f919bb9e4a63ad6d18cab35773bb3a1fef78c356e661f2711da3: Status 404 returned error can't find the container with id e069ed014a89f919bb9e4a63ad6d18cab35773bb3a1fef78c356e661f2711da3
Sep 10 20:38:05.222497 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:05.222429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw" event={"ID":"65148aca-66c0-450c-bdce-5dc0fa70e51c","Type":"ContainerStarted","Data":"e069ed014a89f919bb9e4a63ad6d18cab35773bb3a1fef78c356e661f2711da3"}
Sep 10 20:38:06.230660 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:06.230619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" event={"ID":"d5f12e0c-0398-41b8-8aea-7b5675fe570b","Type":"ContainerStarted","Data":"25dfba3ebaeca6db66cd44fe3f7104770ab4d7a0faea4697f5c5649d3d6a56e9"}
Sep 10 20:38:06.231112 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:06.230726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" event={"ID":"d5f12e0c-0398-41b8-8aea-7b5675fe570b","Type":"ContainerStarted","Data":"c098cd52b277d831e8c049d7c96911de0d369371f2461f41cc9be4ee414ae834"}
Sep 10 20:38:06.231112 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:06.230775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" event={"ID":"d5f12e0c-0398-41b8-8aea-7b5675fe570b","Type":"ContainerStarted","Data":"7e26bcaadb0815d82409a91637e5e74198d485b2b12f830d111438c9d9dbe559"}
Sep 10 20:38:08.241909 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:08.241884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerStarted","Data":"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"}
Sep 10 20:38:08.242232 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:08.241989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerStarted","Data":"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"}
Sep 10 20:38:08.242232 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:08.242027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerStarted","Data":"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"}
Sep 10 20:38:08.242232 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:08.242057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerStarted","Data":"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"}
Sep 10 20:38:08.244511 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:08.244487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" event={"ID":"d5f12e0c-0398-41b8-8aea-7b5675fe570b","Type":"ContainerStarted","Data":"647653b4bb8ee31d41594e86ffaee3607aa78f7f4e26529bd1ee44ad397818b0"}
Sep 10 20:38:08.246398 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:08.246371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw" event={"ID":"65148aca-66c0-450c-bdce-5dc0fa70e51c","Type":"ContainerStarted","Data":"94e90ef7e542c0e6589032dd4dd5c6db34ed3dbc1164bfe5cbde3d6bab085ef2"}
Sep 10 20:38:08.282025 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:08.281984    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw" podStartSLOduration=2.725289965 podStartE2EDuration="5.281967696s" totalImagesPullingTime="2.556677731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:08.280627956 +0000 UTC m=+124.191665254" watchObservedRunningTime="2026-09-10 20:38:08.281967696 +0000 UTC m=+124.193004980"
Sep 10 20:38:09.253249 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:09.253216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" event={"ID":"d5f12e0c-0398-41b8-8aea-7b5675fe570b","Type":"ContainerStarted","Data":"0497c235c8bf8cb40ec6cfa4a0f13eda4f5e50b04a795510ac28d3e9a45edb90"}
Sep 10 20:38:09.253852 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:09.253296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" event={"ID":"d5f12e0c-0398-41b8-8aea-7b5675fe570b","Type":"ContainerStarted","Data":"8aac2d5c72f8fed4375663dc506e2785263c9726d08920594bb183913b3794be"}
Sep 10 20:38:09.253852 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:09.253345    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:09.256428 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:09.256402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerStarted","Data":"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"}
Sep 10 20:38:09.256538 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:09.256457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerStarted","Data":"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"}
Sep 10 20:38:09.297225 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:09.297187    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8" podStartSLOduration=2.659425564 podStartE2EDuration="8.297175242s" totalImagesPullingTime="5.637749678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:38:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:09.295232478 +0000 UTC m=+125.206269759" watchObservedRunningTime="2026-09-10 20:38:09.297175242 +0000 UTC m=+125.208212526"
Sep 10 20:38:09.341124 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:09.341085    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.935985649 podStartE2EDuration="9.341071257s" totalImagesPullingTime="5.385326921s" totalInitContainerRuntime="19.758687ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:38:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:09.338602256 +0000 UTC m=+125.249639541" watchObservedRunningTime="2026-09-10 20:38:09.341071257 +0000 UTC m=+125.252108541"
Sep 10 20:38:10.266893 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:10.266860    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f746ccff4-z8sh8"
Sep 10 20:38:13.289536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:13.289500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:38:13.291861 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:13.291839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9171816d-e76b-42c2-bb56-e341aaed3db4-metrics-certs\") pod \"network-metrics-daemon-kjnmq\" (UID: \"9171816d-e76b-42c2-bb56-e341aaed3db4\") " pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:38:13.306132 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:13.306108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ks6z6\""
Sep 10 20:38:13.313108 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:13.313088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnmq"
Sep 10 20:38:13.647744 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:13.647720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kjnmq"]
Sep 10 20:38:13.650343 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:38:13.650312    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9171816d_e76b_42c2_bb56_e341aaed3db4.slice/crio-9ff7762585f39051a17cd56ca7a331cbbd355776f204f03cf0d5727a9c19b152 WatchSource:0}: Error finding container 9ff7762585f39051a17cd56ca7a331cbbd355776f204f03cf0d5727a9c19b152: Status 404 returned error can't find the container with id 9ff7762585f39051a17cd56ca7a331cbbd355776f204f03cf0d5727a9c19b152
Sep 10 20:38:14.277381 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:14.277344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kjnmq" event={"ID":"9171816d-e76b-42c2-bb56-e341aaed3db4","Type":"ContainerStarted","Data":"9ff7762585f39051a17cd56ca7a331cbbd355776f204f03cf0d5727a9c19b152"}
Sep 10 20:38:15.285483 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:15.285457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kjnmq" event={"ID":"9171816d-e76b-42c2-bb56-e341aaed3db4","Type":"ContainerStarted","Data":"98a9f7dc2025d43283c5e60ae4cdee4723418f1d463e3e502a2dc22be191a63f"}
Sep 10 20:38:16.291379 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:16.291347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kjnmq" event={"ID":"9171816d-e76b-42c2-bb56-e341aaed3db4","Type":"ContainerStarted","Data":"e26dae7d5af2c90eaec18ca00ebd65afcae31bf277665e9f102c6805556edaca"}
Sep 10 20:38:16.318050 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:16.318009    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kjnmq" podStartSLOduration=130.954937222 podStartE2EDuration="2m12.31799312s" totalImagesPullingTime="1.363055898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:16.317009033 +0000 UTC m=+132.228046316" watchObservedRunningTime="2026-09-10 20:38:16.31799312 +0000 UTC m=+132.229030426"
Sep 10 20:38:19.305944 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:19.305873    2565 generic.go:356] "Generic (PLEG): container finished" podID="3992e74c-f2ea-432a-8b36-bdfe17d9766a" containerID="3481526ac0413601b7d61be4cd82c4995531d8f2e33f45126b0a021e330a812e" exitCode=0
Sep 10 20:38:19.306249 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:19.305948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-w5d4j" event={"ID":"3992e74c-f2ea-432a-8b36-bdfe17d9766a","Type":"ContainerDied","Data":"3481526ac0413601b7d61be4cd82c4995531d8f2e33f45126b0a021e330a812e"}
Sep 10 20:38:19.306289 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:19.306271    2565 scope.go:117] "RemoveContainer" containerID="3481526ac0413601b7d61be4cd82c4995531d8f2e33f45126b0a021e330a812e"
Sep 10 20:38:20.311874 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:20.311839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-w5d4j" event={"ID":"3992e74c-f2ea-432a-8b36-bdfe17d9766a","Type":"ContainerStarted","Data":"4c8d3473f349361cc0ca57e220183764fe29ca6fbf726e49cd553d91f84d87ca"}
Sep 10 20:38:20.319156 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:20.319125    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bp5xz_dbfc039d-4acd-4649-a477-b18f2c799a86/serve-healthcheck-canary/0.log"
Sep 10 20:38:22.190173 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.190126    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8f65f69d5-qtxfc" podUID="6afa84a1-0bbc-487b-81ed-3e9405e8b245" containerName="console" containerID="cri-o://de88ed6784fe99b4b5b7348c665c4632dfa986f71d13984857aa08b7bd60ba0e" gracePeriod=15
Sep 10 20:38:22.321981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.321944    2565 generic.go:356] "Generic (PLEG): container finished" podID="6afa84a1-0bbc-487b-81ed-3e9405e8b245" containerID="de88ed6784fe99b4b5b7348c665c4632dfa986f71d13984857aa08b7bd60ba0e" exitCode=0
Sep 10 20:38:22.322104 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.322016    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f65f69d5-qtxfc" event={"ID":"6afa84a1-0bbc-487b-81ed-3e9405e8b245","Type":"ContainerDied","Data":"de88ed6784fe99b4b5b7348c665c4632dfa986f71d13984857aa08b7bd60ba0e"}
Sep 10 20:38:22.432887 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.432865    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:38:22.462835 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.462775    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhl28\" (UniqueName: \"kubernetes.io/projected/6afa84a1-0bbc-487b-81ed-3e9405e8b245-kube-api-access-zhl28\") pod \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") "
Sep 10 20:38:22.462950 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.462841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-oauth-serving-cert\") pod \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") "
Sep 10 20:38:22.462950 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.462868    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-service-ca\") pod \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") "
Sep 10 20:38:22.462950 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.462897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-config\") pod \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") "
Sep 10 20:38:22.462950 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.462920    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-oauth-config\") pod \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") "
Sep 10 20:38:22.463148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.462998    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-serving-cert\") pod \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\" (UID: \"6afa84a1-0bbc-487b-81ed-3e9405e8b245\") "
Sep 10 20:38:22.463306 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.463272    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-config" (OuterVolumeSpecName: "console-config") pod "6afa84a1-0bbc-487b-81ed-3e9405e8b245" (UID: "6afa84a1-0bbc-487b-81ed-3e9405e8b245"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:38:22.463412 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.463280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-service-ca" (OuterVolumeSpecName: "service-ca") pod "6afa84a1-0bbc-487b-81ed-3e9405e8b245" (UID: "6afa84a1-0bbc-487b-81ed-3e9405e8b245"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:38:22.463412 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.463285    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6afa84a1-0bbc-487b-81ed-3e9405e8b245" (UID: "6afa84a1-0bbc-487b-81ed-3e9405e8b245"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:38:22.465073 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.465049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afa84a1-0bbc-487b-81ed-3e9405e8b245-kube-api-access-zhl28" (OuterVolumeSpecName: "kube-api-access-zhl28") pod "6afa84a1-0bbc-487b-81ed-3e9405e8b245" (UID: "6afa84a1-0bbc-487b-81ed-3e9405e8b245"). InnerVolumeSpecName "kube-api-access-zhl28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:38:22.465214 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.465189    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6afa84a1-0bbc-487b-81ed-3e9405e8b245" (UID: "6afa84a1-0bbc-487b-81ed-3e9405e8b245"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:38:22.465280 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.465208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6afa84a1-0bbc-487b-81ed-3e9405e8b245" (UID: "6afa84a1-0bbc-487b-81ed-3e9405e8b245"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:38:22.564376 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.564340    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-oauth-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:38:22.564376 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.564372    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:38:22.564376 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.564383    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:38:22.564545 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.564392    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-oauth-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:38:22.564545 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.564400    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa84a1-0bbc-487b-81ed-3e9405e8b245-console-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:38:22.564545 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:22.564410    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhl28\" (UniqueName: \"kubernetes.io/projected/6afa84a1-0bbc-487b-81ed-3e9405e8b245-kube-api-access-zhl28\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.328301 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:23.328273    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f65f69d5-qtxfc"
Sep 10 20:38:23.328735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:23.328235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f65f69d5-qtxfc" event={"ID":"6afa84a1-0bbc-487b-81ed-3e9405e8b245","Type":"ContainerDied","Data":"51858bed67b6739ddc581217caca74396ceb3de3cfa2fca30473263b220bb321"}
Sep 10 20:38:23.328735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:23.328459    2565 scope.go:117] "RemoveContainer" containerID="de88ed6784fe99b4b5b7348c665c4632dfa986f71d13984857aa08b7bd60ba0e"
Sep 10 20:38:23.348665 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:23.348639    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f65f69d5-qtxfc"]
Sep 10 20:38:23.351925 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:23.351900    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8f65f69d5-qtxfc"]
Sep 10 20:38:24.128143 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:24.128110    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:24.128315 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:24.128175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:24.589860 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:24.589826    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afa84a1-0bbc-487b-81ed-3e9405e8b245" path="/var/lib/kubelet/pods/6afa84a1-0bbc-487b-81ed-3e9405e8b245/volumes"
Sep 10 20:38:44.133381 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:44.133348    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:38:44.137167 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:38:44.137145    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74b7d9f5dc-d28pw"
Sep 10 20:39:19.867063 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:19.867027    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:39:19.867549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:19.867311    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="alertmanager" containerID="cri-o://6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595" gracePeriod=120
Sep 10 20:39:19.867549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:19.867378    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-web" containerID="cri-o://cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453" gracePeriod=120
Sep 10 20:39:19.867549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:19.867399    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="config-reloader" containerID="cri-o://a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e" gracePeriod=120
Sep 10 20:39:19.867549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:19.867430    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="prom-label-proxy" containerID="cri-o://c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3" gracePeriod=120
Sep 10 20:39:19.867549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:19.867378    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-metric" containerID="cri-o://57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80" gracePeriod=120
Sep 10 20:39:19.867549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:19.867483    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy" containerID="cri-o://82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a" gracePeriod=120
Sep 10 20:39:20.557847 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.557815    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerID="c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3" exitCode=0
Sep 10 20:39:20.557847 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.557843    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerID="82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a" exitCode=0
Sep 10 20:39:20.557847 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.557854    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerID="a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e" exitCode=0
Sep 10 20:39:20.558077 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.557861    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerID="6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595" exitCode=0
Sep 10 20:39:20.558077 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.557882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"}
Sep 10 20:39:20.558077 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.557948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"}
Sep 10 20:39:20.558077 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.557975    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"}
Sep 10 20:39:20.558077 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:20.558000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"}
Sep 10 20:39:21.125003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.124981    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.205097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.204971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-trusted-ca-bundle\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-metrics-client-ca\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-config-volume\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205192    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-main-db\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205221    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205256    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-cluster-tls-config\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-web-config\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-main-tls\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205357    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-config-out\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99xwb\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-kube-api-access-99xwb\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-tls-assets\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205446    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205460    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205472    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-web\") pod \"ef3d1682-647e-4a10-b610-62ea4936ed89\" (UID: \"ef3d1682-647e-4a10-b610-62ea4936ed89\") "
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205509    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:39:21.205575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205528    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:39:21.206049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205760    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.206049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205781    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef3d1682-647e-4a10-b610-62ea4936ed89-metrics-client-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.206049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.205796    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-alertmanager-main-db\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.208706 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.208454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-kube-api-access-99xwb" (OuterVolumeSpecName: "kube-api-access-99xwb") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "kube-api-access-99xwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:39:21.208843 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.208673    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:39:21.209056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.208957    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:39:21.209207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.209134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:39:21.209404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.209327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-config-volume" (OuterVolumeSpecName: "config-volume") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:39:21.209610 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.209538    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:39:21.210433 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.210402    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-config-out" (OuterVolumeSpecName: "config-out") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:39:21.210835 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.210817    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:39:21.213028 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.212940    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:39:21.220448 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.220422    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-web-config" (OuterVolumeSpecName: "web-config") pod "ef3d1682-647e-4a10-b610-62ea4936ed89" (UID: "ef3d1682-647e-4a10-b610-62ea4936ed89"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:39:21.306947 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306918    2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-config-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.306947 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306944    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306955    2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-cluster-tls-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306964    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-web-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306973    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-main-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306981    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef3d1682-647e-4a10-b610-62ea4936ed89-config-out\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306990    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xwb\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-kube-api-access-99xwb\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.306997    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef3d1682-647e-4a10-b610-62ea4936ed89-tls-assets\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.307005    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.307096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.307014    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef3d1682-647e-4a10-b610-62ea4936ed89-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:39:21.564838 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.564755    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerID="57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80" exitCode=0
Sep 10 20:39:21.564838 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.564778    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerID="cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453" exitCode=0
Sep 10 20:39:21.565047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.564841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"}
Sep 10 20:39:21.565047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.564865    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.565047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.564943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"}
Sep 10 20:39:21.565047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.564977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef3d1682-647e-4a10-b610-62ea4936ed89","Type":"ContainerDied","Data":"75786f19353a3c7788db2ca355a40dce8037ae7050d80462ac18e25bd731c321"}
Sep 10 20:39:21.565047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.565009    2565 scope.go:117] "RemoveContainer" containerID="c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"
Sep 10 20:39:21.573110 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.573060    2565 scope.go:117] "RemoveContainer" containerID="57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"
Sep 10 20:39:21.580321 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.580298    2565 scope.go:117] "RemoveContainer" containerID="82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"
Sep 10 20:39:21.586993 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.586979    2565 scope.go:117] "RemoveContainer" containerID="cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"
Sep 10 20:39:21.591579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.591559    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:39:21.594175 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.594146    2565 scope.go:117] "RemoveContainer" containerID="a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"
Sep 10 20:39:21.596604 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.596582    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:39:21.601058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.601042    2565 scope.go:117] "RemoveContainer" containerID="6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"
Sep 10 20:39:21.607797 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.607775    2565 scope.go:117] "RemoveContainer" containerID="7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8"
Sep 10 20:39:21.614350 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.614334    2565 scope.go:117] "RemoveContainer" containerID="c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"
Sep 10 20:39:21.614603 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:39:21.614584    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3\": container with ID starting with c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3 not found: ID does not exist" containerID="c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"
Sep 10 20:39:21.614648 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.614611    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"} err="failed to get container status \"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3\": rpc error: code = NotFound desc = could not find container \"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3\": container with ID starting with c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3 not found: ID does not exist"
Sep 10 20:39:21.614648 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.614644    2565 scope.go:117] "RemoveContainer" containerID="57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"
Sep 10 20:39:21.614911 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:39:21.614893    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80\": container with ID starting with 57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80 not found: ID does not exist" containerID="57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"
Sep 10 20:39:21.614954 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.614917    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"} err="failed to get container status \"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80\": rpc error: code = NotFound desc = could not find container \"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80\": container with ID starting with 57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80 not found: ID does not exist"
Sep 10 20:39:21.614954 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.614933    2565 scope.go:117] "RemoveContainer" containerID="82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"
Sep 10 20:39:21.615171 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:39:21.615152    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a\": container with ID starting with 82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a not found: ID does not exist" containerID="82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"
Sep 10 20:39:21.615207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615176    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"} err="failed to get container status \"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a\": rpc error: code = NotFound desc = could not find container \"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a\": container with ID starting with 82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a not found: ID does not exist"
Sep 10 20:39:21.615207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615191    2565 scope.go:117] "RemoveContainer" containerID="cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"
Sep 10 20:39:21.615417 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:39:21.615401    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453\": container with ID starting with cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453 not found: ID does not exist" containerID="cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"
Sep 10 20:39:21.615471 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615421    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"} err="failed to get container status \"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453\": rpc error: code = NotFound desc = could not find container \"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453\": container with ID starting with cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453 not found: ID does not exist"
Sep 10 20:39:21.615471 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615435    2565 scope.go:117] "RemoveContainer" containerID="a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"
Sep 10 20:39:21.615651 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:39:21.615634    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e\": container with ID starting with a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e not found: ID does not exist" containerID="a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"
Sep 10 20:39:21.615707 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615655    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"} err="failed to get container status \"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e\": rpc error: code = NotFound desc = could not find container \"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e\": container with ID starting with a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e not found: ID does not exist"
Sep 10 20:39:21.615707 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615668    2565 scope.go:117] "RemoveContainer" containerID="6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"
Sep 10 20:39:21.615928 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:39:21.615914    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595\": container with ID starting with 6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595 not found: ID does not exist" containerID="6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"
Sep 10 20:39:21.615970 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615931    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"} err="failed to get container status \"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595\": rpc error: code = NotFound desc = could not find container \"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595\": container with ID starting with 6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595 not found: ID does not exist"
Sep 10 20:39:21.615970 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.615943    2565 scope.go:117] "RemoveContainer" containerID="7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8"
Sep 10 20:39:21.616128 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:39:21.616114    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8\": container with ID starting with 7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8 not found: ID does not exist" containerID="7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8"
Sep 10 20:39:21.616163 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616130    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8"} err="failed to get container status \"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8\": rpc error: code = NotFound desc = could not find container \"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8\": container with ID starting with 7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8 not found: ID does not exist"
Sep 10 20:39:21.616163 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616142    2565 scope.go:117] "RemoveContainer" containerID="c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"
Sep 10 20:39:21.616352 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616335    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3"} err="failed to get container status \"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3\": rpc error: code = NotFound desc = could not find container \"c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3\": container with ID starting with c05a7e91e65c249bef52fac0c177ff60000e189f5a2fe8159fed92224738b1f3 not found: ID does not exist"
Sep 10 20:39:21.616389 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616353    2565 scope.go:117] "RemoveContainer" containerID="57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"
Sep 10 20:39:21.616571 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616554    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80"} err="failed to get container status \"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80\": rpc error: code = NotFound desc = could not find container \"57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80\": container with ID starting with 57f5469cdee0cc522bc819fb3409b68a1090b874d0392bd7db89166db41d8d80 not found: ID does not exist"
Sep 10 20:39:21.616571 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616570    2565 scope.go:117] "RemoveContainer" containerID="82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"
Sep 10 20:39:21.616791 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616776    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a"} err="failed to get container status \"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a\": rpc error: code = NotFound desc = could not find container \"82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a\": container with ID starting with 82da9e57287408fdc5c523e592739ef260c682a131c529516fc117c9c3d9307a not found: ID does not exist"
Sep 10 20:39:21.616847 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616790    2565 scope.go:117] "RemoveContainer" containerID="cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"
Sep 10 20:39:21.616975 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616960    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453"} err="failed to get container status \"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453\": rpc error: code = NotFound desc = could not find container \"cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453\": container with ID starting with cb5b152994ac14efab20a8cfea7c1d7e14cd7c4f9a1d745f46c72e756e334453 not found: ID does not exist"
Sep 10 20:39:21.617026 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.616975    2565 scope.go:117] "RemoveContainer" containerID="a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"
Sep 10 20:39:21.617225 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.617207    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e"} err="failed to get container status \"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e\": rpc error: code = NotFound desc = could not find container \"a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e\": container with ID starting with a61ffafa9b57998e332c6de395c9765ba60738ea9698818ccabcee42973c8b2e not found: ID does not exist"
Sep 10 20:39:21.617314 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.617225    2565 scope.go:117] "RemoveContainer" containerID="6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"
Sep 10 20:39:21.617420 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.617406    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595"} err="failed to get container status \"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595\": rpc error: code = NotFound desc = could not find container \"6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595\": container with ID starting with 6c416158086e34862e598a347c70ca5b341079eb1aaff6d677e389dfe0a00595 not found: ID does not exist"
Sep 10 20:39:21.617460 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.617420    2565 scope.go:117] "RemoveContainer" containerID="7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8"
Sep 10 20:39:21.617650 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.617631    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8"} err="failed to get container status \"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8\": rpc error: code = NotFound desc = could not find container \"7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8\": container with ID starting with 7a1b06b46e0058fadddb27d99a5dd0ce363184f593ab9234bf67e7f136a93bd8 not found: ID does not exist"
Sep 10 20:39:21.632026 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632008    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:39:21.632270 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632260    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="alertmanager"
Sep 10 20:39:21.632316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632273    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="alertmanager"
Sep 10 20:39:21.632316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632293    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="config-reloader"
Sep 10 20:39:21.632316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632302    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="config-reloader"
Sep 10 20:39:21.632316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632312    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-metric"
Sep 10 20:39:21.632316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632317    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-metric"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632324    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="prom-label-proxy"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632329    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="prom-label-proxy"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632339    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afa84a1-0bbc-487b-81ed-3e9405e8b245" containerName="console"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632344    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afa84a1-0bbc-487b-81ed-3e9405e8b245" containerName="console"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632350    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-web"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632355    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-web"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632362    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632366    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632376    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="init-config-reloader"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632381    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="init-config-reloader"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632428    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632435    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="prom-label-proxy"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632443    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-web"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632450    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="kube-rbac-proxy-metric"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632455    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afa84a1-0bbc-487b-81ed-3e9405e8b245" containerName="console"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632460    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="alertmanager"
Sep 10 20:39:21.632476 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.632466    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" containerName="config-reloader"
Sep 10 20:39:21.637007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.636993    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.640919 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.640897    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 20:39:21.641023 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.640977    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 20:39:21.641023 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.641017    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 20:39:21.641164 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.641142    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 20:39:21.641613 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.641358    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9cmlr\""
Sep 10 20:39:21.641613 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.641362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 20:39:21.641613 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.641500    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 20:39:21.643491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.643469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 20:39:21.643612 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.643526    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 20:39:21.647217 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.647201    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 20:39:21.657099 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.657079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:39:21.709527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-config-volume\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vd7\" (UniqueName: \"kubernetes.io/projected/be703d20-4404-44be-a3da-96a045de8bdf-kube-api-access-b9vd7\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be703d20-4404-44be-a3da-96a045de8bdf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be703d20-4404-44be-a3da-96a045de8bdf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be703d20-4404-44be-a3da-96a045de8bdf-config-out\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be703d20-4404-44be-a3da-96a045de8bdf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709946 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-web-config\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709946 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709946 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.709946 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.709879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be703d20-4404-44be-a3da-96a045de8bdf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.810927 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.810899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-web-config\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811035 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.810929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811035 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.810954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811035 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.810979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be703d20-4404-44be-a3da-96a045de8bdf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811035 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.811027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.811055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-config-volume\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.811085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.811106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vd7\" (UniqueName: \"kubernetes.io/projected/be703d20-4404-44be-a3da-96a045de8bdf-kube-api-access-b9vd7\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.811132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be703d20-4404-44be-a3da-96a045de8bdf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.811152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be703d20-4404-44be-a3da-96a045de8bdf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.811243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.811178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.812127 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.812102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be703d20-4404-44be-a3da-96a045de8bdf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.812214 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.812161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be703d20-4404-44be-a3da-96a045de8bdf-config-out\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.812283 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.812210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be703d20-4404-44be-a3da-96a045de8bdf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.812559 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.812532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be703d20-4404-44be-a3da-96a045de8bdf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814150 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814215 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-web-config\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814215 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814352 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814352 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814663 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be703d20-4404-44be-a3da-96a045de8bdf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814731 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be703d20-4404-44be-a3da-96a045de8bdf-config-volume\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.814868 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.814829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be703d20-4404-44be-a3da-96a045de8bdf-config-out\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.815916 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.815898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be703d20-4404-44be-a3da-96a045de8bdf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.820745 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.820727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vd7\" (UniqueName: \"kubernetes.io/projected/be703d20-4404-44be-a3da-96a045de8bdf-kube-api-access-b9vd7\") pod \"alertmanager-main-0\" (UID: \"be703d20-4404-44be-a3da-96a045de8bdf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:21.947812 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:21.947790    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:39:22.253307 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:22.253282    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:39:22.254509 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:39:22.254481    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe703d20_4404_44be_a3da_96a045de8bdf.slice/crio-0a3e2eb587d79c469613dd4d2913fe97aac67fc42e9c6fc12d616902abe08ed9 WatchSource:0}: Error finding container 0a3e2eb587d79c469613dd4d2913fe97aac67fc42e9c6fc12d616902abe08ed9: Status 404 returned error can't find the container with id 0a3e2eb587d79c469613dd4d2913fe97aac67fc42e9c6fc12d616902abe08ed9
Sep 10 20:39:22.570966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:22.570899    2565 generic.go:356] "Generic (PLEG): container finished" podID="be703d20-4404-44be-a3da-96a045de8bdf" containerID="8565907228af804be5c950b558aa9ce0f6f0356be1887281efe70ecdd4622f62" exitCode=0
Sep 10 20:39:22.570966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:22.570942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerDied","Data":"8565907228af804be5c950b558aa9ce0f6f0356be1887281efe70ecdd4622f62"}
Sep 10 20:39:22.571129 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:22.571012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerStarted","Data":"0a3e2eb587d79c469613dd4d2913fe97aac67fc42e9c6fc12d616902abe08ed9"}
Sep 10 20:39:22.589858 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:22.589830    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3d1682-647e-4a10-b610-62ea4936ed89" path="/var/lib/kubelet/pods/ef3d1682-647e-4a10-b610-62ea4936ed89/volumes"
Sep 10 20:39:23.578947 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.578913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerStarted","Data":"49d6c933cd1cdf630a2cb08217a7f5dee99863104bf9b9a607be2cba6436a833"}
Sep 10 20:39:23.579365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.579027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerStarted","Data":"a2b42b814ebf3301edac7a7ad45801582a7585186e47fc96a43b92c648882119"}
Sep 10 20:39:23.579365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.579078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerStarted","Data":"79b7c85b7b84dfd987e80bfb3cc12be2cd15babffca5e7177b35728a403e4ae5"}
Sep 10 20:39:23.579365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.579122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerStarted","Data":"9b5a8488a51e2df48b4dfe67b2a0f3d9cf1698b903adc69ad604fca86a157655"}
Sep 10 20:39:23.579365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.579161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerStarted","Data":"f93e62a82a9728866540e68c84c4a8e2c339d6672917a8bd64e3ff333695ebe4"}
Sep 10 20:39:23.579365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.579198    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be703d20-4404-44be-a3da-96a045de8bdf","Type":"ContainerStarted","Data":"aca46f21b95f420032691dec8a7ec08497204e3bd1cd5e99d3f47566550d6d07"}
Sep 10 20:39:23.624150 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.624095    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.605203628 podStartE2EDuration="2.624082702s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.879074ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:39:23.622210786 +0000 UTC m=+199.533248069" watchObservedRunningTime="2026-09-10 20:39:23.624082702 +0000 UTC m=+199.535119986"
Sep 10 20:39:23.893179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.893144    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f6959d868-jbtbm"]
Sep 10 20:39:23.896843 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.896821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.900241 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.900211    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 20:39:23.900485 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.900467    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 20:39:23.900789 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.900772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 20:39:23.900987 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.900967    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dljvq\""
Sep 10 20:39:23.901266 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.901090    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 20:39:23.901266 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.901143    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 20:39:23.906613 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.906411    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 20:39:23.917969 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.917940    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6959d868-jbtbm"]
Sep 10 20:39:23.927345 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-serving-certs-ca-bundle\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.927430 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.927430 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-secret-telemeter-client\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.927512 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-telemeter-client-tls\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.927512 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpk9\" (UniqueName: \"kubernetes.io/projected/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-kube-api-access-4mpk9\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.927512 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.927615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-federate-client-tls\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:23.927615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:23.927567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-metrics-client-ca\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.027900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.027870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-telemeter-client-tls\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.027900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.027906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpk9\" (UniqueName: \"kubernetes.io/projected/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-kube-api-access-4mpk9\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.027930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.027956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-federate-client-tls\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.027984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-metrics-client-ca\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.028010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-serving-certs-ca-bundle\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.028036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.028060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-secret-telemeter-client\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028873 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.028849    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-metrics-client-ca\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.028981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.028849    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-serving-certs-ca-bundle\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.029206 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.029166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.030473 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.030447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-telemeter-client-tls\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.030555 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.030509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.031003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.030983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-secret-telemeter-client\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.031347 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.031330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-federate-client-tls\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.037446 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.037425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpk9\" (UniqueName: \"kubernetes.io/projected/0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4-kube-api-access-4mpk9\") pod \"telemeter-client-6f6959d868-jbtbm\" (UID: \"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4\") " pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.210303 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.210213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm"
Sep 10 20:39:24.506615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.505672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6959d868-jbtbm"]
Sep 10 20:39:24.508426 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:39:24.507217    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d62fc2d_2d7f_4b34_879a_30ecd5c6dec4.slice/crio-6b6815ea77d711ebc2ce8e9be33e0ba6b5241bc7118b6b1256146b5eaf4890e8 WatchSource:0}: Error finding container 6b6815ea77d711ebc2ce8e9be33e0ba6b5241bc7118b6b1256146b5eaf4890e8: Status 404 returned error can't find the container with id 6b6815ea77d711ebc2ce8e9be33e0ba6b5241bc7118b6b1256146b5eaf4890e8
Sep 10 20:39:24.593744 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:24.593714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm" event={"ID":"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4","Type":"ContainerStarted","Data":"6b6815ea77d711ebc2ce8e9be33e0ba6b5241bc7118b6b1256146b5eaf4890e8"}
Sep 10 20:39:26.595357 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.595328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm" event={"ID":"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4","Type":"ContainerStarted","Data":"3f4e6136c9caa2a4602f2a350f2521ed3ef39bdfab8cfd8fd75a11d7fa7a9de2"}
Sep 10 20:39:26.595817 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.595376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm" event={"ID":"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4","Type":"ContainerStarted","Data":"286e6b5d594ce7d4d51e1345b35bc40e3cf43dd6185d7bbb9ee17d9800ea8a14"}
Sep 10 20:39:26.595817 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.595399    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm" event={"ID":"0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4","Type":"ContainerStarted","Data":"c9e3e4614ec41acdf7240a281937438f6d9f6531efbcdcfc864712a5fe0e297f"}
Sep 10 20:39:26.630653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.630604    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f6959d868-jbtbm" podStartSLOduration=1.929349043 podStartE2EDuration="3.630588983s" totalImagesPullingTime="1.70123994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:39:26.627789306 +0000 UTC m=+202.538826590" watchObservedRunningTime="2026-09-10 20:39:26.630588983 +0000 UTC m=+202.541626266"
Sep 10 20:39:26.770591 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.770554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c74bc7dc4-8q4c8"]
Sep 10 20:39:26.774047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.774030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.783606 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.783586    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 20:39:26.784142 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.784123    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 20:39:26.784237 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.784169    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 20:39:26.784237 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.784217    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 20:39:26.784602 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.784579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lrjhl\""
Sep 10 20:39:26.785327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.785309    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 20:39:26.789465 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.789449    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 20:39:26.803324 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.800325    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c74bc7dc4-8q4c8"]
Sep 10 20:39:26.849562 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.849496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq76f\" (UniqueName: \"kubernetes.io/projected/b51307c0-10ce-4465-9cd6-30eb6f2292d6-kube-api-access-vq76f\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.849562 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.849535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-service-ca\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.849714 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.849595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-oauth-config\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.849714 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.849636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-config\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.849796 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.849715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-trusted-ca-bundle\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.849796 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.849755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-serving-cert\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.849796 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.849781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-oauth-serving-cert\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.950764 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.950734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq76f\" (UniqueName: \"kubernetes.io/projected/b51307c0-10ce-4465-9cd6-30eb6f2292d6-kube-api-access-vq76f\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.950951 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.950934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-service-ca\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.951049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.951029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-oauth-config\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.951142 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.951084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-config\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.951142 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.951131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-trusted-ca-bundle\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.951250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.951175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-serving-cert\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.951250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.951210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-oauth-serving-cert\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.951857 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.951828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-service-ca\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.952063 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.952040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-oauth-serving-cert\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.952149 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.952048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-config\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.952430 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.952405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-trusted-ca-bundle\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.953657 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.953631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-oauth-config\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.954175 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.954151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-serving-cert\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:26.961188 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:26.961165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq76f\" (UniqueName: \"kubernetes.io/projected/b51307c0-10ce-4465-9cd6-30eb6f2292d6-kube-api-access-vq76f\") pod \"console-c74bc7dc4-8q4c8\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") " pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:27.082895 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:27.082865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:27.371187 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:27.371153    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c74bc7dc4-8q4c8"]
Sep 10 20:39:27.372627 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:39:27.372600    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51307c0_10ce_4465_9cd6_30eb6f2292d6.slice/crio-2925af8df0d1aa662473f83df542f90083a1f04e3533d4782ead16e928d3fcd0 WatchSource:0}: Error finding container 2925af8df0d1aa662473f83df542f90083a1f04e3533d4782ead16e928d3fcd0: Status 404 returned error can't find the container with id 2925af8df0d1aa662473f83df542f90083a1f04e3533d4782ead16e928d3fcd0
Sep 10 20:39:27.601299 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:27.601265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c74bc7dc4-8q4c8" event={"ID":"b51307c0-10ce-4465-9cd6-30eb6f2292d6","Type":"ContainerStarted","Data":"0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a"}
Sep 10 20:39:27.601646 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:27.601319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c74bc7dc4-8q4c8" event={"ID":"b51307c0-10ce-4465-9cd6-30eb6f2292d6","Type":"ContainerStarted","Data":"2925af8df0d1aa662473f83df542f90083a1f04e3533d4782ead16e928d3fcd0"}
Sep 10 20:39:27.625107 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:27.625056    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c74bc7dc4-8q4c8" podStartSLOduration=1.625039101 podStartE2EDuration="1.625039101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:39:27.624940437 +0000 UTC m=+203.535977733" watchObservedRunningTime="2026-09-10 20:39:27.625039101 +0000 UTC m=+203.536076388"
Sep 10 20:39:37.084051 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:37.083995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:37.084051 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:37.084062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:37.089096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:37.089070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:39:37.646725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:39:37.646697    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:41:04.505102 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:04.505070    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:41:04.513560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:04.513535    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:41:04.516124 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:04.516101    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:41:04.520246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:04.520215    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:41:04.520246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:04.520241    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 20:41:29.707743 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.707708    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86bccd8f47-gj9rh"]
Sep 10 20:41:29.711204 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.711181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.723403 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.723377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86bccd8f47-gj9rh"]
Sep 10 20:41:29.792093 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.792064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-trusted-ca-bundle\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.792093 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.792096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-serving-cert\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.792240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.792118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5snx\" (UniqueName: \"kubernetes.io/projected/5fa77224-3386-4d41-b753-a9cd477e5f8b-kube-api-access-c5snx\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.792240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.792203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-oauth-config\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.792240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.792232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-config\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.792334 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.792258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-oauth-serving-cert\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.792334 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.792320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-service-ca\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.893438 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.893412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-service-ca\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.893536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.893457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-trusted-ca-bundle\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.893536 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.893475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-serving-cert\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.893625 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.893603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5snx\" (UniqueName: \"kubernetes.io/projected/5fa77224-3386-4d41-b753-a9cd477e5f8b-kube-api-access-c5snx\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.893676 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.893661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-oauth-config\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.893804 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.893717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-config\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.893804 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.893759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-oauth-serving-cert\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.894171 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.894147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-service-ca\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.894433 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.894405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-config\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.894510 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.894437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-oauth-serving-cert\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.894510 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.894466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-trusted-ca-bundle\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.896091 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.896062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-oauth-config\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.896178 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.896138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-serving-cert\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:29.902977 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:29.902957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5snx\" (UniqueName: \"kubernetes.io/projected/5fa77224-3386-4d41-b753-a9cd477e5f8b-kube-api-access-c5snx\") pod \"console-86bccd8f47-gj9rh\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") " pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:30.020435 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:30.020379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:30.293927 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:30.293884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86bccd8f47-gj9rh"]
Sep 10 20:41:30.296489 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:41:30.296457    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa77224_3386_4d41_b753_a9cd477e5f8b.slice/crio-82f47934052ce0111ec5afcb300b09dd2b9375283cfc57d9e00cd4899c8dba60 WatchSource:0}: Error finding container 82f47934052ce0111ec5afcb300b09dd2b9375283cfc57d9e00cd4899c8dba60: Status 404 returned error can't find the container with id 82f47934052ce0111ec5afcb300b09dd2b9375283cfc57d9e00cd4899c8dba60
Sep 10 20:41:31.104288 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:31.104253    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bccd8f47-gj9rh" event={"ID":"5fa77224-3386-4d41-b753-a9cd477e5f8b","Type":"ContainerStarted","Data":"7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d"}
Sep 10 20:41:31.104644 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:31.104309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bccd8f47-gj9rh" event={"ID":"5fa77224-3386-4d41-b753-a9cd477e5f8b","Type":"ContainerStarted","Data":"82f47934052ce0111ec5afcb300b09dd2b9375283cfc57d9e00cd4899c8dba60"}
Sep 10 20:41:31.126444 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:31.126394    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86bccd8f47-gj9rh" podStartSLOduration=2.126379405 podStartE2EDuration="2.126379405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:41:31.124904442 +0000 UTC m=+327.035941863" watchObservedRunningTime="2026-09-10 20:41:31.126379405 +0000 UTC m=+327.037416735"
Sep 10 20:41:40.021352 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:40.021313    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:40.021823 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:40.021377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:40.026068 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:40.026041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:40.146957 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:40.146928    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:41:40.196624 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:41:40.196600    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c74bc7dc4-8q4c8"]
Sep 10 20:42:05.216916 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.216834    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c74bc7dc4-8q4c8" podUID="b51307c0-10ce-4465-9cd6-30eb6f2292d6" containerName="console" containerID="cri-o://0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a" gracePeriod=15
Sep 10 20:42:05.456245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.456221    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:42:05.566362 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566294    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-oauth-serving-cert\") pod \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") "
Sep 10 20:42:05.566362 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-trusted-ca-bundle\") pod \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") "
Sep 10 20:42:05.566552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566369    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-service-ca\") pod \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") "
Sep 10 20:42:05.566552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-serving-cert\") pod \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") "
Sep 10 20:42:05.566552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566417    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq76f\" (UniqueName: \"kubernetes.io/projected/b51307c0-10ce-4465-9cd6-30eb6f2292d6-kube-api-access-vq76f\") pod \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") "
Sep 10 20:42:05.566552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566441    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-config\") pod \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") "
Sep 10 20:42:05.566552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566495    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-oauth-config\") pod \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\" (UID: \"b51307c0-10ce-4465-9cd6-30eb6f2292d6\") "
Sep 10 20:42:05.566819 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b51307c0-10ce-4465-9cd6-30eb6f2292d6" (UID: "b51307c0-10ce-4465-9cd6-30eb6f2292d6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:42:05.566925 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-service-ca" (OuterVolumeSpecName: "service-ca") pod "b51307c0-10ce-4465-9cd6-30eb6f2292d6" (UID: "b51307c0-10ce-4465-9cd6-30eb6f2292d6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:42:05.567006 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566982    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-config" (OuterVolumeSpecName: "console-config") pod "b51307c0-10ce-4465-9cd6-30eb6f2292d6" (UID: "b51307c0-10ce-4465-9cd6-30eb6f2292d6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:42:05.567006 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.566995    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b51307c0-10ce-4465-9cd6-30eb6f2292d6" (UID: "b51307c0-10ce-4465-9cd6-30eb6f2292d6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:42:05.568464 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.568442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b51307c0-10ce-4465-9cd6-30eb6f2292d6" (UID: "b51307c0-10ce-4465-9cd6-30eb6f2292d6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:42:05.568581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.568564    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51307c0-10ce-4465-9cd6-30eb6f2292d6-kube-api-access-vq76f" (OuterVolumeSpecName: "kube-api-access-vq76f") pod "b51307c0-10ce-4465-9cd6-30eb6f2292d6" (UID: "b51307c0-10ce-4465-9cd6-30eb6f2292d6"). InnerVolumeSpecName "kube-api-access-vq76f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:42:05.568670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.568650    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b51307c0-10ce-4465-9cd6-30eb6f2292d6" (UID: "b51307c0-10ce-4465-9cd6-30eb6f2292d6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:42:05.667929 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.667899    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-oauth-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:42:05.667929 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.667925    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-trusted-ca-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:42:05.668094 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.667939    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:42:05.668094 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.667951    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:42:05.668094 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.667963    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq76f\" (UniqueName: \"kubernetes.io/projected/b51307c0-10ce-4465-9cd6-30eb6f2292d6-kube-api-access-vq76f\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:42:05.668094 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.667977    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:42:05.668094 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:05.667989    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51307c0-10ce-4465-9cd6-30eb6f2292d6-console-oauth-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:42:06.249740 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.249677    2565 generic.go:356] "Generic (PLEG): container finished" podID="b51307c0-10ce-4465-9cd6-30eb6f2292d6" containerID="0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a" exitCode=0
Sep 10 20:42:06.249740 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.249743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c74bc7dc4-8q4c8"
Sep 10 20:42:06.250194 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.249773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c74bc7dc4-8q4c8" event={"ID":"b51307c0-10ce-4465-9cd6-30eb6f2292d6","Type":"ContainerDied","Data":"0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a"}
Sep 10 20:42:06.250194 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.249836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c74bc7dc4-8q4c8" event={"ID":"b51307c0-10ce-4465-9cd6-30eb6f2292d6","Type":"ContainerDied","Data":"2925af8df0d1aa662473f83df542f90083a1f04e3533d4782ead16e928d3fcd0"}
Sep 10 20:42:06.250194 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.249857    2565 scope.go:117] "RemoveContainer" containerID="0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a"
Sep 10 20:42:06.259845 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.259824    2565 scope.go:117] "RemoveContainer" containerID="0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a"
Sep 10 20:42:06.260157 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:42:06.260137    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a\": container with ID starting with 0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a not found: ID does not exist" containerID="0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a"
Sep 10 20:42:06.260199 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.260167    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a"} err="failed to get container status \"0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a\": rpc error: code = NotFound desc = could not find container \"0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a\": container with ID starting with 0d94a78db98a9af1f6468ade2c78bbc48c42586c754e406dee8949573d7c6d1a not found: ID does not exist"
Sep 10 20:42:06.273207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.273183    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c74bc7dc4-8q4c8"]
Sep 10 20:42:06.274943 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.274921    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c74bc7dc4-8q4c8"]
Sep 10 20:42:06.589855 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:06.589821    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b51307c0-10ce-4465-9cd6-30eb6f2292d6" path="/var/lib/kubelet/pods/b51307c0-10ce-4465-9cd6-30eb6f2292d6/volumes"
Sep 10 20:42:44.069467 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.069429    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c45d6f6dd-lwvjk"]
Sep 10 20:42:44.070095 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.069914    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51307c0-10ce-4465-9cd6-30eb6f2292d6" containerName="console"
Sep 10 20:42:44.070095 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.069934    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51307c0-10ce-4465-9cd6-30eb6f2292d6" containerName="console"
Sep 10 20:42:44.070095 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.070044    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51307c0-10ce-4465-9cd6-30eb6f2292d6" containerName="console"
Sep 10 20:42:44.073491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.073471    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.093201 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.093179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c45d6f6dd-lwvjk"]
Sep 10 20:42:44.152990 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.152961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-trusted-ca-bundle\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.153109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.152994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvjr\" (UniqueName: \"kubernetes.io/projected/9d82c47e-4c69-4b23-b785-3d219141f720-kube-api-access-cwvjr\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.153109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.153011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-service-ca\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.153109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.153028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-oauth-serving-cert\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.153109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.153079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-console-config\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.153109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.153095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d82c47e-4c69-4b23-b785-3d219141f720-console-serving-cert\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.153345 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.153197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d82c47e-4c69-4b23-b785-3d219141f720-console-oauth-config\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.253775 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.253751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-oauth-serving-cert\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.253872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.253787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-console-config\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.253872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.253803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d82c47e-4c69-4b23-b785-3d219141f720-console-serving-cert\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.253872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.253842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d82c47e-4c69-4b23-b785-3d219141f720-console-oauth-config\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.254020 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.253874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-trusted-ca-bundle\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.254020 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.253892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvjr\" (UniqueName: \"kubernetes.io/projected/9d82c47e-4c69-4b23-b785-3d219141f720-kube-api-access-cwvjr\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.254020 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.253908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-service-ca\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.254583 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.254555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-oauth-serving-cert\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.254703 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.254555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-console-config\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.254703 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.254652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-service-ca\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.254825 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.254760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d82c47e-4c69-4b23-b785-3d219141f720-trusted-ca-bundle\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.256450 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.256424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d82c47e-4c69-4b23-b785-3d219141f720-console-oauth-config\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.256545 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.256501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d82c47e-4c69-4b23-b785-3d219141f720-console-serving-cert\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.262995 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.262972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvjr\" (UniqueName: \"kubernetes.io/projected/9d82c47e-4c69-4b23-b785-3d219141f720-kube-api-access-cwvjr\") pod \"console-c45d6f6dd-lwvjk\" (UID: \"9d82c47e-4c69-4b23-b785-3d219141f720\") " pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.382753 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.382727    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:44.658320 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:44.658292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c45d6f6dd-lwvjk"]
Sep 10 20:42:44.661258 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:42:44.661211    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d82c47e_4c69_4b23_b785_3d219141f720.slice/crio-51919f0ea31171a8848613e5347ebfcc66b7ebe6c4d9d30ed7e3cc4a01382cb8 WatchSource:0}: Error finding container 51919f0ea31171a8848613e5347ebfcc66b7ebe6c4d9d30ed7e3cc4a01382cb8: Status 404 returned error can't find the container with id 51919f0ea31171a8848613e5347ebfcc66b7ebe6c4d9d30ed7e3cc4a01382cb8
Sep 10 20:42:45.422182 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:45.422146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c45d6f6dd-lwvjk" event={"ID":"9d82c47e-4c69-4b23-b785-3d219141f720","Type":"ContainerStarted","Data":"a2b587b3d5ea4aedaf5f46c2d2299500254f772cdd3c432510ff16a9f60cf794"}
Sep 10 20:42:45.422642 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:45.422193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c45d6f6dd-lwvjk" event={"ID":"9d82c47e-4c69-4b23-b785-3d219141f720","Type":"ContainerStarted","Data":"51919f0ea31171a8848613e5347ebfcc66b7ebe6c4d9d30ed7e3cc4a01382cb8"}
Sep 10 20:42:45.442488 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:45.442434    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c45d6f6dd-lwvjk" podStartSLOduration=1.4424165549999999 podStartE2EDuration="1.442416555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:42:45.440662801 +0000 UTC m=+401.351700085" watchObservedRunningTime="2026-09-10 20:42:45.442416555 +0000 UTC m=+401.353453838"
Sep 10 20:42:54.383743 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:54.383699    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:54.384271 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:54.383792    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:54.388674 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:54.388647    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:54.465780 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:54.465754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c45d6f6dd-lwvjk"
Sep 10 20:42:54.515973 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:42:54.515940    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86bccd8f47-gj9rh"]
Sep 10 20:43:19.536151 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.536101    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86bccd8f47-gj9rh" podUID="5fa77224-3386-4d41-b753-a9cd477e5f8b" containerName="console" containerID="cri-o://7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d" gracePeriod=15
Sep 10 20:43:19.773527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.773501    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:43:19.832012 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.831950    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-trusted-ca-bundle\") pod \"5fa77224-3386-4d41-b753-a9cd477e5f8b\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") "
Sep 10 20:43:19.832012 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.831999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-service-ca\") pod \"5fa77224-3386-4d41-b753-a9cd477e5f8b\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") "
Sep 10 20:43:19.832190 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-serving-cert\") pod \"5fa77224-3386-4d41-b753-a9cd477e5f8b\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") "
Sep 10 20:43:19.832190 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832051    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5snx\" (UniqueName: \"kubernetes.io/projected/5fa77224-3386-4d41-b753-a9cd477e5f8b-kube-api-access-c5snx\") pod \"5fa77224-3386-4d41-b753-a9cd477e5f8b\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") "
Sep 10 20:43:19.832190 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-config\") pod \"5fa77224-3386-4d41-b753-a9cd477e5f8b\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") "
Sep 10 20:43:19.832190 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832095    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-oauth-config\") pod \"5fa77224-3386-4d41-b753-a9cd477e5f8b\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") "
Sep 10 20:43:19.832190 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832123    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-oauth-serving-cert\") pod \"5fa77224-3386-4d41-b753-a9cd477e5f8b\" (UID: \"5fa77224-3386-4d41-b753-a9cd477e5f8b\") "
Sep 10 20:43:19.832502 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832465    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5fa77224-3386-4d41-b753-a9cd477e5f8b" (UID: "5fa77224-3386-4d41-b753-a9cd477e5f8b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:43:19.832622 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832523    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-config" (OuterVolumeSpecName: "console-config") pod "5fa77224-3386-4d41-b753-a9cd477e5f8b" (UID: "5fa77224-3386-4d41-b753-a9cd477e5f8b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:43:19.832622 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832555    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5fa77224-3386-4d41-b753-a9cd477e5f8b" (UID: "5fa77224-3386-4d41-b753-a9cd477e5f8b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:43:19.832622 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.832573    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-service-ca" (OuterVolumeSpecName: "service-ca") pod "5fa77224-3386-4d41-b753-a9cd477e5f8b" (UID: "5fa77224-3386-4d41-b753-a9cd477e5f8b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:43:19.834165 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.834144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5fa77224-3386-4d41-b753-a9cd477e5f8b" (UID: "5fa77224-3386-4d41-b753-a9cd477e5f8b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:43:19.834245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.834196    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa77224-3386-4d41-b753-a9cd477e5f8b-kube-api-access-c5snx" (OuterVolumeSpecName: "kube-api-access-c5snx") pod "5fa77224-3386-4d41-b753-a9cd477e5f8b" (UID: "5fa77224-3386-4d41-b753-a9cd477e5f8b"). InnerVolumeSpecName "kube-api-access-c5snx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:43:19.834309 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.834281    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5fa77224-3386-4d41-b753-a9cd477e5f8b" (UID: "5fa77224-3386-4d41-b753-a9cd477e5f8b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:43:19.933549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.933524    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-oauth-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:43:19.933549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.933548    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-oauth-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:43:19.933711 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.933561    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-trusted-ca-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:43:19.933711 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.933574    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:43:19.933711 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.933588    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:43:19.933711 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.933601    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5snx\" (UniqueName: \"kubernetes.io/projected/5fa77224-3386-4d41-b753-a9cd477e5f8b-kube-api-access-c5snx\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:43:19.933711 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:19.933613    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fa77224-3386-4d41-b753-a9cd477e5f8b-console-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:43:20.569590 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.569552    2565 generic.go:356] "Generic (PLEG): container finished" podID="5fa77224-3386-4d41-b753-a9cd477e5f8b" containerID="7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d" exitCode=0
Sep 10 20:43:20.570038 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.569604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bccd8f47-gj9rh"
Sep 10 20:43:20.570038 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.569612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bccd8f47-gj9rh" event={"ID":"5fa77224-3386-4d41-b753-a9cd477e5f8b","Type":"ContainerDied","Data":"7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d"}
Sep 10 20:43:20.570038 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.569660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bccd8f47-gj9rh" event={"ID":"5fa77224-3386-4d41-b753-a9cd477e5f8b","Type":"ContainerDied","Data":"82f47934052ce0111ec5afcb300b09dd2b9375283cfc57d9e00cd4899c8dba60"}
Sep 10 20:43:20.570038 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.569710    2565 scope.go:117] "RemoveContainer" containerID="7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d"
Sep 10 20:43:20.579096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.579075    2565 scope.go:117] "RemoveContainer" containerID="7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d"
Sep 10 20:43:20.579384 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:43:20.579363    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d\": container with ID starting with 7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d not found: ID does not exist" containerID="7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d"
Sep 10 20:43:20.579449 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.579392    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d"} err="failed to get container status \"7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d\": rpc error: code = NotFound desc = could not find container \"7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d\": container with ID starting with 7ba6c9be1f811990d95c23251060fbec910ec0734e6b8627b6a0628df0ee4e0d not found: ID does not exist"
Sep 10 20:43:20.592599 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.592573    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86bccd8f47-gj9rh"]
Sep 10 20:43:20.599014 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:20.598994    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86bccd8f47-gj9rh"]
Sep 10 20:43:22.589298 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:22.589265    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa77224-3386-4d41-b753-a9cd477e5f8b" path="/var/lib/kubelet/pods/5fa77224-3386-4d41-b753-a9cd477e5f8b/volumes"
Sep 10 20:43:51.618879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.618847    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"]
Sep 10 20:43:51.619356 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.619168    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa77224-3386-4d41-b753-a9cd477e5f8b" containerName="console"
Sep 10 20:43:51.619356 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.619178    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa77224-3386-4d41-b753-a9cd477e5f8b" containerName="console"
Sep 10 20:43:51.619356 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.619237    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa77224-3386-4d41-b753-a9cd477e5f8b" containerName="console"
Sep 10 20:43:51.621904 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.621887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:51.624958 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.624939    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 20:43:51.626397 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.626381    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-shjn5\""
Sep 10 20:43:51.626478 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.626404    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 20:43:51.633336 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.633311    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"]
Sep 10 20:43:51.760469 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.760445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wttrs\" (UniqueName: \"kubernetes.io/projected/29392f89-aa5e-4bd7-9b18-f5232fb5f1c6-kube-api-access-wttrs\") pod \"cert-manager-cainjector-559cdd8c87-pz6r5\" (UID: \"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:51.760599 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.760481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29392f89-aa5e-4bd7-9b18-f5232fb5f1c6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pz6r5\" (UID: \"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:51.861652 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.861628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttrs\" (UniqueName: \"kubernetes.io/projected/29392f89-aa5e-4bd7-9b18-f5232fb5f1c6-kube-api-access-wttrs\") pod \"cert-manager-cainjector-559cdd8c87-pz6r5\" (UID: \"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:51.861784 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.861663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29392f89-aa5e-4bd7-9b18-f5232fb5f1c6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pz6r5\" (UID: \"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:51.871920 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.871868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29392f89-aa5e-4bd7-9b18-f5232fb5f1c6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pz6r5\" (UID: \"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:51.872188 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.872165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttrs\" (UniqueName: \"kubernetes.io/projected/29392f89-aa5e-4bd7-9b18-f5232fb5f1c6-kube-api-access-wttrs\") pod \"cert-manager-cainjector-559cdd8c87-pz6r5\" (UID: \"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:51.930654 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:51.930622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"
Sep 10 20:43:52.199877 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:52.199853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5"]
Sep 10 20:43:52.201432 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:43:52.201404    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29392f89_aa5e_4bd7_9b18_f5232fb5f1c6.slice/crio-98b60ec0a956497886890afd55c80764ae4c16c125d6288e1b62ebf935134b03 WatchSource:0}: Error finding container 98b60ec0a956497886890afd55c80764ae4c16c125d6288e1b62ebf935134b03: Status 404 returned error can't find the container with id 98b60ec0a956497886890afd55c80764ae4c16c125d6288e1b62ebf935134b03
Sep 10 20:43:52.203489 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:52.203474    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:43:52.699504 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:52.699475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5" event={"ID":"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6","Type":"ContainerStarted","Data":"98b60ec0a956497886890afd55c80764ae4c16c125d6288e1b62ebf935134b03"}
Sep 10 20:43:55.714700 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:55.714657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5" event={"ID":"29392f89-aa5e-4bd7-9b18-f5232fb5f1c6","Type":"ContainerStarted","Data":"41fc145043a118ba05bcdf0ee8f77a887a6bf6fc85f2c49d482d7c63fea449cc"}
Sep 10 20:43:55.734752 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:55.734705    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pz6r5" podStartSLOduration=1.7788888360000001 podStartE2EDuration="4.734674294s" totalImagesPullingTime="2.955785458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:43:55.733402736 +0000 UTC m=+471.644440018" watchObservedRunningTime="2026-09-10 20:43:55.734674294 +0000 UTC m=+471.645711578"
Sep 10 20:43:59.936961 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:59.936921    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"]
Sep 10 20:43:59.940309 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:59.940284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:43:59.943441 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:59.943419    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:43:59.945018 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:59.944991    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 20:43:59.945120 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:59.944991    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7ml7p\""
Sep 10 20:43:59.951542 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:43:59.951522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"]
Sep 10 20:44:00.027950 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.027909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8ct\" (UniqueName: \"kubernetes.io/projected/2051e58e-f44c-4860-8fbb-7102ca1c1414-kube-api-access-8d8ct\") pod \"openshift-lws-operator-fd8ccff4c-j48m6\" (UID: \"2051e58e-f44c-4860-8fbb-7102ca1c1414\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:44:00.027950 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.027948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2051e58e-f44c-4860-8fbb-7102ca1c1414-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j48m6\" (UID: \"2051e58e-f44c-4860-8fbb-7102ca1c1414\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:44:00.129049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.129013    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8ct\" (UniqueName: \"kubernetes.io/projected/2051e58e-f44c-4860-8fbb-7102ca1c1414-kube-api-access-8d8ct\") pod \"openshift-lws-operator-fd8ccff4c-j48m6\" (UID: \"2051e58e-f44c-4860-8fbb-7102ca1c1414\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:44:00.129240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.129053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2051e58e-f44c-4860-8fbb-7102ca1c1414-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j48m6\" (UID: \"2051e58e-f44c-4860-8fbb-7102ca1c1414\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:44:00.129471 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.129447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2051e58e-f44c-4860-8fbb-7102ca1c1414-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j48m6\" (UID: \"2051e58e-f44c-4860-8fbb-7102ca1c1414\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:44:00.138640 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.138616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8ct\" (UniqueName: \"kubernetes.io/projected/2051e58e-f44c-4860-8fbb-7102ca1c1414-kube-api-access-8d8ct\") pod \"openshift-lws-operator-fd8ccff4c-j48m6\" (UID: \"2051e58e-f44c-4860-8fbb-7102ca1c1414\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:44:00.248921 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.248858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"
Sep 10 20:44:00.522463 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.522441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6"]
Sep 10 20:44:00.524449 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:44:00.524423    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2051e58e_f44c_4860_8fbb_7102ca1c1414.slice/crio-d6d4ee2cd4616e70342f59b6be1e5bbe234dec56ae3399ce6aa0ee23053833a6 WatchSource:0}: Error finding container d6d4ee2cd4616e70342f59b6be1e5bbe234dec56ae3399ce6aa0ee23053833a6: Status 404 returned error can't find the container with id d6d4ee2cd4616e70342f59b6be1e5bbe234dec56ae3399ce6aa0ee23053833a6
Sep 10 20:44:00.736114 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:00.736086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6" event={"ID":"2051e58e-f44c-4860-8fbb-7102ca1c1414","Type":"ContainerStarted","Data":"d6d4ee2cd4616e70342f59b6be1e5bbe234dec56ae3399ce6aa0ee23053833a6"}
Sep 10 20:44:03.753082 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:03.753046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6" event={"ID":"2051e58e-f44c-4860-8fbb-7102ca1c1414","Type":"ContainerStarted","Data":"50fe49753818de300f7fa64f4b6efc36704a67b3524d17b989d3ca4b613d76ab"}
Sep 10 20:44:03.772072 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:03.772026    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j48m6" podStartSLOduration=2.221456802 podStartE2EDuration="4.772010012s" totalImagesPullingTime="2.55055321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:44:03.770426794 +0000 UTC m=+479.681464078" watchObservedRunningTime="2026-09-10 20:44:03.772010012 +0000 UTC m=+479.683047296"
Sep 10 20:44:31.203935 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.203902    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"]
Sep 10 20:44:31.207358 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.207340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.211895 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.211870    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 20:44:31.212025 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.211895    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 20:44:31.212615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.212599    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v6fqb\""
Sep 10 20:44:31.212615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.212608    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 20:44:31.212760 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.212610    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 20:44:31.239586 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.239564    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"]
Sep 10 20:44:31.266106 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.266083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2kn\" (UniqueName: \"kubernetes.io/projected/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-kube-api-access-8r2kn\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.266211 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.266145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.266211 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.266195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-manifests\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.266299 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.266230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-webhook-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.266299 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.266256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-tmp\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.366601 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.366578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2kn\" (UniqueName: \"kubernetes.io/projected/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-kube-api-access-8r2kn\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.366734 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.366628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.366734 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.366664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-manifests\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.366734 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.366729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-webhook-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.366905 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.366757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-tmp\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.367109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.367085    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-manifests\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.369021 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.368994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-tmp\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.369244 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.369227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-webhook-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.369312 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.369283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.378990 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.378969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2kn\" (UniqueName: \"kubernetes.io/projected/2e2ccf4b-2f6d-46fa-bfe0-b18c41007505-kube-api-access-8r2kn\") pod \"opendatahub-operator-controller-manager-59df7859b6-xm2qs\" (UID: \"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.517275 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.517223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:31.798134 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.798106    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"]
Sep 10 20:44:31.798568 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:44:31.798548    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2ccf4b_2f6d_46fa_bfe0_b18c41007505.slice/crio-3d905894e636d6354555e8a7a3475be705e00a3f49d847b6d3152390e4be5e5e WatchSource:0}: Error finding container 3d905894e636d6354555e8a7a3475be705e00a3f49d847b6d3152390e4be5e5e: Status 404 returned error can't find the container with id 3d905894e636d6354555e8a7a3475be705e00a3f49d847b6d3152390e4be5e5e
Sep 10 20:44:31.879432 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:31.879406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs" event={"ID":"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505","Type":"ContainerStarted","Data":"3d905894e636d6354555e8a7a3475be705e00a3f49d847b6d3152390e4be5e5e"}
Sep 10 20:44:34.895254 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:34.895226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs" event={"ID":"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505","Type":"ContainerStarted","Data":"66260851c162c8abf690f36cba84e0b4d85f4b284521f5d0d5e393099936b966"}
Sep 10 20:44:36.906585 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:36.906550    2565 generic.go:356] "Generic (PLEG): container finished" podID="2e2ccf4b-2f6d-46fa-bfe0-b18c41007505" containerID="66260851c162c8abf690f36cba84e0b4d85f4b284521f5d0d5e393099936b966" exitCode=0
Sep 10 20:44:36.907082 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:36.906604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs" event={"ID":"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505","Type":"ContainerDied","Data":"66260851c162c8abf690f36cba84e0b4d85f4b284521f5d0d5e393099936b966"}
Sep 10 20:44:37.916593 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:37.913147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs" event={"ID":"2e2ccf4b-2f6d-46fa-bfe0-b18c41007505","Type":"ContainerStarted","Data":"c2fd984227f234261f1ac48ec1536de19e4987515bc2286d09e97b4c843dd7c4"}
Sep 10 20:44:37.916593 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:37.914480    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:37.943065 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:37.943019    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs" podStartSLOduration=2.575535325 podStartE2EDuration="6.943006211s" totalImagesPullingTime="3.315701069s" totalInitContainerRuntime="1.051769817s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:44:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:44:37.941029816 +0000 UTC m=+513.852067127" watchObservedRunningTime="2026-09-10 20:44:37.943006211 +0000 UTC m=+513.854043495"
Sep 10 20:44:48.920462 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:48.920424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-xm2qs"
Sep 10 20:44:50.925055 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.925026    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"]
Sep 10 20:44:50.937983 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.937958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:50.941262 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.941241    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 20:44:50.941493 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.941478    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 20:44:50.942995 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.942971    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 20:44:50.943102 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.943023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9gchw\""
Sep 10 20:44:50.943102 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.943029    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 20:44:50.947031 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:50.947008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"]
Sep 10 20:44:51.022130 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.022108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tls-certs\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.022245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.022147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tmp\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.022311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.022249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqk5\" (UniqueName: \"kubernetes.io/projected/4de93ecd-2faf-4b03-9db2-f6cbb829a253-kube-api-access-7zqk5\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.122893 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.122867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tmp\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.122984 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.122928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqk5\" (UniqueName: \"kubernetes.io/projected/4de93ecd-2faf-4b03-9db2-f6cbb829a253-kube-api-access-7zqk5\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.122984 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.122957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tls-certs\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.123094 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:44:51.123046    2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 20:44:51.123146 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:44:51.123105    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tls-certs podName:4de93ecd-2faf-4b03-9db2-f6cbb829a253 nodeName:}" failed. No retries permitted until 2026-09-10 20:44:51.623090039 +0000 UTC m=+527.534127301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tls-certs") pod "kube-auth-proxy-556d9c8688-r45j2" (UID: "4de93ecd-2faf-4b03-9db2-f6cbb829a253") : secret "kube-auth-proxy-tls" not found
Sep 10 20:44:51.125098 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.125077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tmp\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.136537 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.136506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqk5\" (UniqueName: \"kubernetes.io/projected/4de93ecd-2faf-4b03-9db2-f6cbb829a253-kube-api-access-7zqk5\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.626814 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.626643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tls-certs\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.629075 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.629040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de93ecd-2faf-4b03-9db2-f6cbb829a253-tls-certs\") pod \"kube-auth-proxy-556d9c8688-r45j2\" (UID: \"4de93ecd-2faf-4b03-9db2-f6cbb829a253\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:51.849003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:51.848977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"
Sep 10 20:44:52.133026 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.132996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-556d9c8688-r45j2"]
Sep 10 20:44:52.134877 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:44:52.134850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de93ecd_2faf_4b03_9db2_f6cbb829a253.slice/crio-ff756d5514b87f8808aa62e5bde8003d52a55e3a78f8c4a4e21d77f6ee326237 WatchSource:0}: Error finding container ff756d5514b87f8808aa62e5bde8003d52a55e3a78f8c4a4e21d77f6ee326237: Status 404 returned error can't find the container with id ff756d5514b87f8808aa62e5bde8003d52a55e3a78f8c4a4e21d77f6ee326237
Sep 10 20:44:52.383113 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.383082    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"]
Sep 10 20:44:52.387921 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.387904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.390933 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.390915    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 20:44:52.391209 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.391187    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 20:44:52.391323 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.391297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 20:44:52.391439 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.391391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 20:44:52.391505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.391483    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8krxv\""
Sep 10 20:44:52.404157 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.404134    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"]
Sep 10 20:44:52.431242 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.431221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.431332 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.431257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.431332 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.431284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.431400 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.431364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.431475 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.431400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.431475 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.431430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgqp\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-kube-api-access-2fgqp\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.431475 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.431452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2927387f-c0eb-47af-a992-29c37da1d342-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.532756 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.532732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.532840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.532769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.532840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.532789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.532840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.532814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgqp\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-kube-api-access-2fgqp\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.532840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.532834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2927387f-c0eb-47af-a992-29c37da1d342-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.533013 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.532869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.533013 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.532894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.533460 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.533436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.535126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.535087    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2927387f-c0eb-47af-a992-29c37da1d342-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.535329 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.535303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.535387 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.535373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.535456 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.535435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.542167 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.542138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgqp\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-kube-api-access-2fgqp\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.542596 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.542577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xtnsb\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.697423 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.697355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:52.982025 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.981942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2" event={"ID":"4de93ecd-2faf-4b03-9db2-f6cbb829a253","Type":"ContainerStarted","Data":"ff756d5514b87f8808aa62e5bde8003d52a55e3a78f8c4a4e21d77f6ee326237"}
Sep 10 20:44:52.989658 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:52.989634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"]
Sep 10 20:44:52.992385 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:44:52.992353    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2927387f_c0eb_47af_a992_29c37da1d342.slice/crio-e9e4697ed8787795ec2760d5503d0502b002f3a2d4c5fe8172a16617a9368284 WatchSource:0}: Error finding container e9e4697ed8787795ec2760d5503d0502b002f3a2d4c5fe8172a16617a9368284: Status 404 returned error can't find the container with id e9e4697ed8787795ec2760d5503d0502b002f3a2d4c5fe8172a16617a9368284
Sep 10 20:44:53.989570 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:53.989529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb" event={"ID":"2927387f-c0eb-47af-a992-29c37da1d342","Type":"ContainerStarted","Data":"e9e4697ed8787795ec2760d5503d0502b002f3a2d4c5fe8172a16617a9368284"}
Sep 10 20:44:54.997995 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:54.997956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2" event={"ID":"4de93ecd-2faf-4b03-9db2-f6cbb829a253","Type":"ContainerStarted","Data":"adfbeeabbb2f0eef8f883456770af8c5d2bac8c3cdb54682cc51bbe1459e3839"}
Sep 10 20:44:55.019573 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:55.019509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-556d9c8688-r45j2" podStartSLOduration=2.374638002 podStartE2EDuration="5.019486368s" totalImagesPullingTime="2.644848366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:44:55.01646383 +0000 UTC m=+530.927501114" watchObservedRunningTime="2026-09-10 20:44:55.019486368 +0000 UTC m=+530.930523653"
Sep 10 20:44:56.527108 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:56.527063    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:44:56.527396 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:56.527125    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:44:57.011525 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:57.011370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb" event={"ID":"2927387f-c0eb-47af-a992-29c37da1d342","Type":"ContainerStarted","Data":"097f8047738fcc627890fc0bd054c24da9d7610c6f03ed5c0d8c7c284165df03"}
Sep 10 20:44:57.011726 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:57.011585    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:44:57.013126 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:57.013101    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-xtnsb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 20:44:57.013219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:57.013158    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb" podUID="2927387f-c0eb-47af-a992-29c37da1d342" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 20:44:57.038426 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:57.038360    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb" podStartSLOduration=1.506352625 podStartE2EDuration="5.038337277s" totalImagesPullingTime="3.531984652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:44:57.035047645 +0000 UTC m=+532.946084942" watchObservedRunningTime="2026-09-10 20:44:57.038337277 +0000 UTC m=+532.949374588"
Sep 10 20:44:58.016584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:44:58.016556    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:45:12.745315 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.745252    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"]
Sep 10 20:45:12.750169 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.750147    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:12.756791 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.756501    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hfrw7\""
Sep 10 20:45:12.756791 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.756661    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 20:45:12.759832 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.759810    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"]
Sep 10 20:45:12.762823 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.762805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:12.766332 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.766297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tx5xk\""
Sep 10 20:45:12.767306 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.767283    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"]
Sep 10 20:45:12.769318 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.769200    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 20:45:12.779532 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.779504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"]
Sep 10 20:45:12.910308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.910280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ff9a55-515f-42ff-98c3-b8ada9d6225e-cert\") pod \"kserve-controller-manager-d6f4b65b8-fqgdq\" (UID: \"36ff9a55-515f-42ff-98c3-b8ada9d6225e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:12.910463 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.910317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfr6v\" (UniqueName: \"kubernetes.io/projected/9910bb83-a139-4222-afa0-d3bee0b2b885-kube-api-access-jfr6v\") pod \"llmisvc-controller-manager-777c6bd7d5-xsvmm\" (UID: \"9910bb83-a139-4222-afa0-d3bee0b2b885\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:12.910463 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.910339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9910bb83-a139-4222-afa0-d3bee0b2b885-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xsvmm\" (UID: \"9910bb83-a139-4222-afa0-d3bee0b2b885\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:12.910463 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:12.910412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gs6\" (UniqueName: \"kubernetes.io/projected/36ff9a55-515f-42ff-98c3-b8ada9d6225e-kube-api-access-89gs6\") pod \"kserve-controller-manager-d6f4b65b8-fqgdq\" (UID: \"36ff9a55-515f-42ff-98c3-b8ada9d6225e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:13.011482 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.011415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ff9a55-515f-42ff-98c3-b8ada9d6225e-cert\") pod \"kserve-controller-manager-d6f4b65b8-fqgdq\" (UID: \"36ff9a55-515f-42ff-98c3-b8ada9d6225e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:13.011482 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.011454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfr6v\" (UniqueName: \"kubernetes.io/projected/9910bb83-a139-4222-afa0-d3bee0b2b885-kube-api-access-jfr6v\") pod \"llmisvc-controller-manager-777c6bd7d5-xsvmm\" (UID: \"9910bb83-a139-4222-afa0-d3bee0b2b885\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:13.011482 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.011474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9910bb83-a139-4222-afa0-d3bee0b2b885-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xsvmm\" (UID: \"9910bb83-a139-4222-afa0-d3bee0b2b885\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:13.011668 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.011497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gs6\" (UniqueName: \"kubernetes.io/projected/36ff9a55-515f-42ff-98c3-b8ada9d6225e-kube-api-access-89gs6\") pod \"kserve-controller-manager-d6f4b65b8-fqgdq\" (UID: \"36ff9a55-515f-42ff-98c3-b8ada9d6225e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:13.011668 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:45:13.011596    2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 10 20:45:13.011765 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:45:13.011720    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9910bb83-a139-4222-afa0-d3bee0b2b885-cert podName:9910bb83-a139-4222-afa0-d3bee0b2b885 nodeName:}" failed. No retries permitted until 2026-09-10 20:45:13.511698983 +0000 UTC m=+549.422736263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9910bb83-a139-4222-afa0-d3bee0b2b885-cert") pod "llmisvc-controller-manager-777c6bd7d5-xsvmm" (UID: "9910bb83-a139-4222-afa0-d3bee0b2b885") : secret "llmisvc-webhook-server-cert" not found
Sep 10 20:45:13.013817 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.013791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ff9a55-515f-42ff-98c3-b8ada9d6225e-cert\") pod \"kserve-controller-manager-d6f4b65b8-fqgdq\" (UID: \"36ff9a55-515f-42ff-98c3-b8ada9d6225e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:13.036096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.036070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfr6v\" (UniqueName: \"kubernetes.io/projected/9910bb83-a139-4222-afa0-d3bee0b2b885-kube-api-access-jfr6v\") pod \"llmisvc-controller-manager-777c6bd7d5-xsvmm\" (UID: \"9910bb83-a139-4222-afa0-d3bee0b2b885\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:13.058804 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.058784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gs6\" (UniqueName: \"kubernetes.io/projected/36ff9a55-515f-42ff-98c3-b8ada9d6225e-kube-api-access-89gs6\") pod \"kserve-controller-manager-d6f4b65b8-fqgdq\" (UID: \"36ff9a55-515f-42ff-98c3-b8ada9d6225e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:13.060590 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.060572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:13.333154 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.333074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"]
Sep 10 20:45:13.336659 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:45:13.336616    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ff9a55_515f_42ff_98c3_b8ada9d6225e.slice/crio-d6e84339e88806b279a8e12fac31b8668db892fd9865b507fa6b40af43556366 WatchSource:0}: Error finding container d6e84339e88806b279a8e12fac31b8668db892fd9865b507fa6b40af43556366: Status 404 returned error can't find the container with id d6e84339e88806b279a8e12fac31b8668db892fd9865b507fa6b40af43556366
Sep 10 20:45:13.516865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.516831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9910bb83-a139-4222-afa0-d3bee0b2b885-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xsvmm\" (UID: \"9910bb83-a139-4222-afa0-d3bee0b2b885\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:13.519268 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.519245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9910bb83-a139-4222-afa0-d3bee0b2b885-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xsvmm\" (UID: \"9910bb83-a139-4222-afa0-d3bee0b2b885\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:13.673346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.673314    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:13.948780 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:13.948744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"]
Sep 10 20:45:13.950425 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:45:13.950399    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9910bb83_a139_4222_afa0_d3bee0b2b885.slice/crio-22c2497bb5139fda2a5400510f1613d6c0dfc2d1327596903280145a9c18d6a4 WatchSource:0}: Error finding container 22c2497bb5139fda2a5400510f1613d6c0dfc2d1327596903280145a9c18d6a4: Status 404 returned error can't find the container with id 22c2497bb5139fda2a5400510f1613d6c0dfc2d1327596903280145a9c18d6a4
Sep 10 20:45:14.102030 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:14.101993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq" event={"ID":"36ff9a55-515f-42ff-98c3-b8ada9d6225e","Type":"ContainerStarted","Data":"d6e84339e88806b279a8e12fac31b8668db892fd9865b507fa6b40af43556366"}
Sep 10 20:45:14.103406 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:14.103369    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm" event={"ID":"9910bb83-a139-4222-afa0-d3bee0b2b885","Type":"ContainerStarted","Data":"22c2497bb5139fda2a5400510f1613d6c0dfc2d1327596903280145a9c18d6a4"}
Sep 10 20:45:14.716217 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:14.716184    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"]
Sep 10 20:45:14.716452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:14.716426    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb" podUID="2927387f-c0eb-47af-a992-29c37da1d342" containerName="discovery" containerID="cri-o://097f8047738fcc627890fc0bd054c24da9d7610c6f03ed5c0d8c7c284165df03" gracePeriod=30
Sep 10 20:45:15.111307 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.111273    2565 generic.go:356] "Generic (PLEG): container finished" podID="2927387f-c0eb-47af-a992-29c37da1d342" containerID="097f8047738fcc627890fc0bd054c24da9d7610c6f03ed5c0d8c7c284165df03" exitCode=0
Sep 10 20:45:15.111755 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.111344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb" event={"ID":"2927387f-c0eb-47af-a992-29c37da1d342","Type":"ContainerDied","Data":"097f8047738fcc627890fc0bd054c24da9d7610c6f03ed5c0d8c7c284165df03"}
Sep 10 20:45:15.538655 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.538599    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:45:15.634423 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.634393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-cacerts\") pod \"2927387f-c0eb-47af-a992-29c37da1d342\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") "
Sep 10 20:45:15.634587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.634458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-istio-token\") pod \"2927387f-c0eb-47af-a992-29c37da1d342\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") "
Sep 10 20:45:15.634587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.634500    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-kubeconfig\") pod \"2927387f-c0eb-47af-a992-29c37da1d342\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") "
Sep 10 20:45:15.634587 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.634564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2927387f-c0eb-47af-a992-29c37da1d342-local-certs\") pod \"2927387f-c0eb-47af-a992-29c37da1d342\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") "
Sep 10 20:45:15.634792 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.634595    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-dns-cert\") pod \"2927387f-c0eb-47af-a992-29c37da1d342\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") "
Sep 10 20:45:15.634792 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.634628    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fgqp\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-kube-api-access-2fgqp\") pod \"2927387f-c0eb-47af-a992-29c37da1d342\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") "
Sep 10 20:45:15.634792 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.634664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-ca-configmap\") pod \"2927387f-c0eb-47af-a992-29c37da1d342\" (UID: \"2927387f-c0eb-47af-a992-29c37da1d342\") "
Sep 10 20:45:15.635649 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.635608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2927387f-c0eb-47af-a992-29c37da1d342" (UID: "2927387f-c0eb-47af-a992-29c37da1d342"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:45:15.638244 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.638210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2927387f-c0eb-47af-a992-29c37da1d342" (UID: "2927387f-c0eb-47af-a992-29c37da1d342"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:45:15.638364 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.638312    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-cacerts" (OuterVolumeSpecName: "cacerts") pod "2927387f-c0eb-47af-a992-29c37da1d342" (UID: "2927387f-c0eb-47af-a992-29c37da1d342"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:45:15.638521 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.638489    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-istio-token" (OuterVolumeSpecName: "istio-token") pod "2927387f-c0eb-47af-a992-29c37da1d342" (UID: "2927387f-c0eb-47af-a992-29c37da1d342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:45:15.638902 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.638855    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2927387f-c0eb-47af-a992-29c37da1d342-local-certs" (OuterVolumeSpecName: "local-certs") pod "2927387f-c0eb-47af-a992-29c37da1d342" (UID: "2927387f-c0eb-47af-a992-29c37da1d342"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:45:15.638902 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.638860    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2927387f-c0eb-47af-a992-29c37da1d342" (UID: "2927387f-c0eb-47af-a992-29c37da1d342"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:45:15.639058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.638893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-kube-api-access-2fgqp" (OuterVolumeSpecName: "kube-api-access-2fgqp") pod "2927387f-c0eb-47af-a992-29c37da1d342" (UID: "2927387f-c0eb-47af-a992-29c37da1d342"). InnerVolumeSpecName "kube-api-access-2fgqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:45:15.736384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.736349    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-istio-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:45:15.736384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.736385    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-kubeconfig\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:45:15.736584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.736400    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2927387f-c0eb-47af-a992-29c37da1d342-local-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:45:15.736584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.736413    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-dns-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:45:15.736584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.736429    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgqp\" (UniqueName: \"kubernetes.io/projected/2927387f-c0eb-47af-a992-29c37da1d342-kube-api-access-2fgqp\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:45:15.736584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.736445    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2927387f-c0eb-47af-a992-29c37da1d342-istio-csr-ca-configmap\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:45:15.736584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:15.736459    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2927387f-c0eb-47af-a992-29c37da1d342-cacerts\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:45:16.117185 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.117167    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"
Sep 10 20:45:16.117517 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.117204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb" event={"ID":"2927387f-c0eb-47af-a992-29c37da1d342","Type":"ContainerDied","Data":"e9e4697ed8787795ec2760d5503d0502b002f3a2d4c5fe8172a16617a9368284"}
Sep 10 20:45:16.117517 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.117263    2565 scope.go:117] "RemoveContainer" containerID="097f8047738fcc627890fc0bd054c24da9d7610c6f03ed5c0d8c7c284165df03"
Sep 10 20:45:16.118904 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.118881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq" event={"ID":"36ff9a55-515f-42ff-98c3-b8ada9d6225e","Type":"ContainerStarted","Data":"9728993196a8ef19a6b62a79ac8623c1200eed0da6a008e6e5391b86b7258a56"}
Sep 10 20:45:16.119042 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.119022    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:16.175960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.175921    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"]
Sep 10 20:45:16.178854 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.178826    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xtnsb"]
Sep 10 20:45:16.221234 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.221158    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq" podStartSLOduration=1.9300338 podStartE2EDuration="4.221142322s" totalImagesPullingTime="2.291108522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:45:16.214456649 +0000 UTC m=+552.125493933" watchObservedRunningTime="2026-09-10 20:45:16.221142322 +0000 UTC m=+552.132179605"
Sep 10 20:45:16.592616 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:16.592579    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2927387f-c0eb-47af-a992-29c37da1d342" path="/var/lib/kubelet/pods/2927387f-c0eb-47af-a992-29c37da1d342/volumes"
Sep 10 20:45:17.124885 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:17.124854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm" event={"ID":"9910bb83-a139-4222-afa0-d3bee0b2b885","Type":"ContainerStarted","Data":"02c9650ba421e991ef63802b9a9d157834269a06e7dd369a486a8c6de7c938e7"}
Sep 10 20:45:17.125247 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:17.124938    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:45:17.221550 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:17.221474    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm" podStartSLOduration=2.250648853 podStartE2EDuration="5.221459003s" totalImagesPullingTime="2.97081015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:45:17.219674349 +0000 UTC m=+553.130711632" watchObservedRunningTime="2026-09-10 20:45:17.221459003 +0000 UTC m=+553.132496331"
Sep 10 20:45:47.131906 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:47.131874    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fqgdq"
Sep 10 20:45:48.139452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:45:48.139423    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xsvmm"
Sep 10 20:46:04.541927 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:04.541898    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:46:04.545769 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:04.545747    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:46:04.546069 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:04.546050    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:46:04.549615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:04.549596    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:46:30.906735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.906706    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"]
Sep 10 20:46:30.908985 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.907060    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2927387f-c0eb-47af-a992-29c37da1d342" containerName="discovery"
Sep 10 20:46:30.908985 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.907072    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2927387f-c0eb-47af-a992-29c37da1d342" containerName="discovery"
Sep 10 20:46:30.908985 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.907146    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2927387f-c0eb-47af-a992-29c37da1d342" containerName="discovery"
Sep 10 20:46:30.910010 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.909994    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"
Sep 10 20:46:30.913290 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.913270    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 20:46:30.913436 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.913414    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 20:46:30.913506 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.913494    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 20:46:30.914774 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.914757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fx7dg\""
Sep 10 20:46:30.917109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.917087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcff2\" (UniqueName: \"kubernetes.io/projected/22f65693-edbc-47a6-8f45-c4938702a9d1-kube-api-access-vcff2\") pod \"dns-operator-controller-manager-7b84cd7c9c-mzt4v\" (UID: \"22f65693-edbc-47a6-8f45-c4938702a9d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"
Sep 10 20:46:30.922545 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:30.922517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"]
Sep 10 20:46:31.017894 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:31.017868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcff2\" (UniqueName: \"kubernetes.io/projected/22f65693-edbc-47a6-8f45-c4938702a9d1-kube-api-access-vcff2\") pod \"dns-operator-controller-manager-7b84cd7c9c-mzt4v\" (UID: \"22f65693-edbc-47a6-8f45-c4938702a9d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"
Sep 10 20:46:31.026807 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:31.026786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcff2\" (UniqueName: \"kubernetes.io/projected/22f65693-edbc-47a6-8f45-c4938702a9d1-kube-api-access-vcff2\") pod \"dns-operator-controller-manager-7b84cd7c9c-mzt4v\" (UID: \"22f65693-edbc-47a6-8f45-c4938702a9d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"
Sep 10 20:46:31.219898 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:31.219831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"
Sep 10 20:46:31.491782 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:31.491754    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"]
Sep 10 20:46:31.493463 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:31.493437    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f65693_edbc_47a6_8f45_c4938702a9d1.slice/crio-ed5b2c8826db5f34aae8c82bee40e47f0d266f1f8dc4eb252d4d1e97c3a589c5 WatchSource:0}: Error finding container ed5b2c8826db5f34aae8c82bee40e47f0d266f1f8dc4eb252d4d1e97c3a589c5: Status 404 returned error can't find the container with id ed5b2c8826db5f34aae8c82bee40e47f0d266f1f8dc4eb252d4d1e97c3a589c5
Sep 10 20:46:32.495654 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:32.495602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v" event={"ID":"22f65693-edbc-47a6-8f45-c4938702a9d1","Type":"ContainerStarted","Data":"ed5b2c8826db5f34aae8c82bee40e47f0d266f1f8dc4eb252d4d1e97c3a589c5"}
Sep 10 20:46:33.527617 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.527576    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rdsz7"]
Sep 10 20:46:33.539300 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.539276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7"
Sep 10 20:46:33.543471 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.543439    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xprjr\""
Sep 10 20:46:33.559285 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.559263    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rdsz7"]
Sep 10 20:46:33.638108 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.638076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxs8\" (UniqueName: \"kubernetes.io/projected/d6cef623-e76d-46ec-9620-20759029568e-kube-api-access-jhxs8\") pod \"authorino-operator-76b478bd78-rdsz7\" (UID: \"d6cef623-e76d-46ec-9620-20759029568e\") " pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7"
Sep 10 20:46:33.738701 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.738658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxs8\" (UniqueName: \"kubernetes.io/projected/d6cef623-e76d-46ec-9620-20759029568e-kube-api-access-jhxs8\") pod \"authorino-operator-76b478bd78-rdsz7\" (UID: \"d6cef623-e76d-46ec-9620-20759029568e\") " pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7"
Sep 10 20:46:33.752553 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.752527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxs8\" (UniqueName: \"kubernetes.io/projected/d6cef623-e76d-46ec-9620-20759029568e-kube-api-access-jhxs8\") pod \"authorino-operator-76b478bd78-rdsz7\" (UID: \"d6cef623-e76d-46ec-9620-20759029568e\") " pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7"
Sep 10 20:46:33.850701 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:33.850654    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7"
Sep 10 20:46:34.210889 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:34.210859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rdsz7"]
Sep 10 20:46:34.211964 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:34.211939    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cef623_e76d_46ec_9620_20759029568e.slice/crio-71ea4b2c8c1174674cb004e789fd20c9a0d746099618afa2fdc61b568e6b5306 WatchSource:0}: Error finding container 71ea4b2c8c1174674cb004e789fd20c9a0d746099618afa2fdc61b568e6b5306: Status 404 returned error can't find the container with id 71ea4b2c8c1174674cb004e789fd20c9a0d746099618afa2fdc61b568e6b5306
Sep 10 20:46:34.508164 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:34.508080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7" event={"ID":"d6cef623-e76d-46ec-9620-20759029568e","Type":"ContainerStarted","Data":"71ea4b2c8c1174674cb004e789fd20c9a0d746099618afa2fdc61b568e6b5306"}
Sep 10 20:46:34.509844 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:34.509814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v" event={"ID":"22f65693-edbc-47a6-8f45-c4938702a9d1","Type":"ContainerStarted","Data":"210b93e2a821d40f81dba6a1e5ccd15396afed91b019013cd90203f8a047683e"}
Sep 10 20:46:34.510056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:34.510041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"
Sep 10 20:46:34.535879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:34.535841    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v" podStartSLOduration=2.030820649 podStartE2EDuration="4.535826954s" totalImagesPullingTime="2.505006305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:34.534939378 +0000 UTC m=+630.445976661" watchObservedRunningTime="2026-09-10 20:46:34.535826954 +0000 UTC m=+630.446864237"
Sep 10 20:46:35.828246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:35.828209    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"]
Sep 10 20:46:35.832408 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:35.832379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:35.835556 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:35.835520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s2whb\""
Sep 10 20:46:35.849733 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:35.849709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"]
Sep 10 20:46:35.857367 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:35.857339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj79z\" (UniqueName: \"kubernetes.io/projected/32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0-kube-api-access-hj79z\") pod \"limitador-operator-controller-manager-74495c69f8-wh2td\" (UID: \"32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:35.958876 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:35.958836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj79z\" (UniqueName: \"kubernetes.io/projected/32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0-kube-api-access-hj79z\") pod \"limitador-operator-controller-manager-74495c69f8-wh2td\" (UID: \"32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:35.972840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:35.972813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj79z\" (UniqueName: \"kubernetes.io/projected/32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0-kube-api-access-hj79z\") pod \"limitador-operator-controller-manager-74495c69f8-wh2td\" (UID: \"32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:36.146739 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:36.146713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:36.418710 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:36.418669    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"]
Sep 10 20:46:36.420110 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:36.420084    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32acb98e_18d3_4cb2_b9eb_d6cfedc4b8f0.slice/crio-41168118302e7d5acbaa55b94497f2f5e18d1057278c2506c992fdf104699a84 WatchSource:0}: Error finding container 41168118302e7d5acbaa55b94497f2f5e18d1057278c2506c992fdf104699a84: Status 404 returned error can't find the container with id 41168118302e7d5acbaa55b94497f2f5e18d1057278c2506c992fdf104699a84
Sep 10 20:46:36.522848 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:36.522818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td" event={"ID":"32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0","Type":"ContainerStarted","Data":"41168118302e7d5acbaa55b94497f2f5e18d1057278c2506c992fdf104699a84"}
Sep 10 20:46:36.524499 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:36.524473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7" event={"ID":"d6cef623-e76d-46ec-9620-20759029568e","Type":"ContainerStarted","Data":"4b8ebc314fe147aa40fa336470f68532b44f3be884c1c6623009533eea0f0eab"}
Sep 10 20:46:36.524609 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:36.524595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7"
Sep 10 20:46:36.554235 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:36.554182    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7" podStartSLOduration=1.7584298889999999 podStartE2EDuration="3.554167554s" totalImagesPullingTime="1.795737665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:36.551579371 +0000 UTC m=+632.462616655" watchObservedRunningTime="2026-09-10 20:46:36.554167554 +0000 UTC m=+632.465204837"
Sep 10 20:46:38.538871 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:38.538835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td" event={"ID":"32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0","Type":"ContainerStarted","Data":"6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d"}
Sep 10 20:46:38.539224 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:38.539017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:38.560991 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:38.560944    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td" podStartSLOduration=1.902507693 podStartE2EDuration="3.560930825s" totalImagesPullingTime="1.658423132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:38.559723111 +0000 UTC m=+634.470760395" watchObservedRunningTime="2026-09-10 20:46:38.560930825 +0000 UTC m=+634.471968108"
Sep 10 20:46:39.739315 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.739281    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"]
Sep 10 20:46:39.742830 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.742814    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:39.746063 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.746044    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jzjqd\""
Sep 10 20:46:39.754919 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.754898    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"]
Sep 10 20:46:39.788879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.788855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9dn\" (UniqueName: \"kubernetes.io/projected/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-kube-api-access-sg9dn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:39.789001 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.788910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:39.889751 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.889726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:39.889842 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.889782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9dn\" (UniqueName: \"kubernetes.io/projected/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-kube-api-access-sg9dn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:39.890077 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.890058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:39.898632 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:39.898608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9dn\" (UniqueName: \"kubernetes.io/projected/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-kube-api-access-sg9dn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:40.052860 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:40.052793    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:40.329468 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:40.329442    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"]
Sep 10 20:46:40.330646 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:40.330619    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e83ea4d_d4fd_4415_8b5c_0ac968de9871.slice/crio-5245716f2817121ecb9f88f6cb75390b5bc60dc1a47f6c0d2a75166de4d5ca76 WatchSource:0}: Error finding container 5245716f2817121ecb9f88f6cb75390b5bc60dc1a47f6c0d2a75166de4d5ca76: Status 404 returned error can't find the container with id 5245716f2817121ecb9f88f6cb75390b5bc60dc1a47f6c0d2a75166de4d5ca76
Sep 10 20:46:40.549444 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:40.549412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7" event={"ID":"4e83ea4d-d4fd-4415-8b5c-0ac968de9871","Type":"ContainerStarted","Data":"5245716f2817121ecb9f88f6cb75390b5bc60dc1a47f6c0d2a75166de4d5ca76"}
Sep 10 20:46:44.572346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:44.572247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7" event={"ID":"4e83ea4d-d4fd-4415-8b5c-0ac968de9871","Type":"ContainerStarted","Data":"52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb"}
Sep 10 20:46:44.572346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:44.572334    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:44.599779 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:44.599735    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7" podStartSLOduration=1.803510421 podStartE2EDuration="5.599721662s" totalImagesPullingTime="3.796211241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:44.595851341 +0000 UTC m=+640.506888624" watchObservedRunningTime="2026-09-10 20:46:44.599721662 +0000 UTC m=+640.510758946"
Sep 10 20:46:45.518200 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:45.518169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-mzt4v"
Sep 10 20:46:46.781267 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.781237    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"]
Sep 10 20:46:46.785054 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.785033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.788194 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.788172    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5j9jh\""
Sep 10 20:46:46.788458 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.788444    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 20:46:46.788590 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.788573    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 20:46:46.794457 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.794436    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"]
Sep 10 20:46:46.853007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.852984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57wz\" (UniqueName: \"kubernetes.io/projected/378d55bb-2fcb-4da8-ab7c-76906014b4c6-kube-api-access-m57wz\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.853119 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.853037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/378d55bb-2fcb-4da8-ab7c-76906014b4c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.853119 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.853075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/378d55bb-2fcb-4da8-ab7c-76906014b4c6-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.953817 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.953783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57wz\" (UniqueName: \"kubernetes.io/projected/378d55bb-2fcb-4da8-ab7c-76906014b4c6-kube-api-access-m57wz\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.953945 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.953853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/378d55bb-2fcb-4da8-ab7c-76906014b4c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.953945 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.953881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/378d55bb-2fcb-4da8-ab7c-76906014b4c6-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.954052 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:46.953984    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 20:46:46.954052 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:46.954051    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378d55bb-2fcb-4da8-ab7c-76906014b4c6-plugin-serving-cert podName:378d55bb-2fcb-4da8-ab7c-76906014b4c6 nodeName:}" failed. No retries permitted until 2026-09-10 20:46:47.454037192 +0000 UTC m=+643.365074454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/378d55bb-2fcb-4da8-ab7c-76906014b4c6-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-bx6fh" (UID: "378d55bb-2fcb-4da8-ab7c-76906014b4c6") : secret "plugin-serving-cert" not found
Sep 10 20:46:46.954511 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.954490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/378d55bb-2fcb-4da8-ab7c-76906014b4c6-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:46.968217 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:46.968197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57wz\" (UniqueName: \"kubernetes.io/projected/378d55bb-2fcb-4da8-ab7c-76906014b4c6-kube-api-access-m57wz\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:47.457979 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:47.457929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/378d55bb-2fcb-4da8-ab7c-76906014b4c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:47.460490 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:47.460462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/378d55bb-2fcb-4da8-ab7c-76906014b4c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bx6fh\" (UID: \"378d55bb-2fcb-4da8-ab7c-76906014b4c6\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:47.532754 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:47.532727    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-rdsz7"
Sep 10 20:46:47.694951 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:47.694922    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"
Sep 10 20:46:47.968851 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:47.968830    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh"]
Sep 10 20:46:47.970628 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:47.970603    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod378d55bb_2fcb_4da8_ab7c_76906014b4c6.slice/crio-589f7d08f03fb8d51f0e1806ca4c4147254868ac637112540d23d6eccf16b83c WatchSource:0}: Error finding container 589f7d08f03fb8d51f0e1806ca4c4147254868ac637112540d23d6eccf16b83c: Status 404 returned error can't find the container with id 589f7d08f03fb8d51f0e1806ca4c4147254868ac637112540d23d6eccf16b83c
Sep 10 20:46:48.595585 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:48.595548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh" event={"ID":"378d55bb-2fcb-4da8-ab7c-76906014b4c6","Type":"ContainerStarted","Data":"589f7d08f03fb8d51f0e1806ca4c4147254868ac637112540d23d6eccf16b83c"}
Sep 10 20:46:49.546157 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:49.546120    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:54.628156 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:54.628119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh" event={"ID":"378d55bb-2fcb-4da8-ab7c-76906014b4c6","Type":"ContainerStarted","Data":"d76da74d94724171e4890d25796adddb53bbe2224369ec67a96236da415d88cc"}
Sep 10 20:46:54.651103 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:54.651057    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bx6fh" podStartSLOduration=2.661455892 podStartE2EDuration="8.651044559s" totalImagesPullingTime="5.989588667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:54.650540449 +0000 UTC m=+650.561577734" watchObservedRunningTime="2026-09-10 20:46:54.651044559 +0000 UTC m=+650.562081842"
Sep 10 20:46:55.580057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:55.580024    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:56.915960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:56.915924    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"]
Sep 10 20:46:56.920785 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:56.920765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:56.935780 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:56.935755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"]
Sep 10 20:46:57.040544 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.040515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjjv\" (UniqueName: \"kubernetes.io/projected/80e3527b-f167-4892-81e0-139aed21aa9e-kube-api-access-pjjjv\") pod \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.040655 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.040615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80e3527b-f167-4892-81e0-139aed21aa9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.140959 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.140937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80e3527b-f167-4892-81e0-139aed21aa9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.141057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.140975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjjv\" (UniqueName: \"kubernetes.io/projected/80e3527b-f167-4892-81e0-139aed21aa9e-kube-api-access-pjjjv\") pod \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.141280 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.141262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80e3527b-f167-4892-81e0-139aed21aa9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.150152 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.150124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjjv\" (UniqueName: \"kubernetes.io/projected/80e3527b-f167-4892-81e0-139aed21aa9e-kube-api-access-pjjjv\") pod \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.231143 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.231084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.510456 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.510432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"]
Sep 10 20:46:57.512075 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:57.512041    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e3527b_f167_4892_81e0_139aed21aa9e.slice/crio-14942445f7cb1a6f5fd5f5068f53017273d40cadfec1e35f12d51aa4ca1b7853 WatchSource:0}: Error finding container 14942445f7cb1a6f5fd5f5068f53017273d40cadfec1e35f12d51aa4ca1b7853: Status 404 returned error can't find the container with id 14942445f7cb1a6f5fd5f5068f53017273d40cadfec1e35f12d51aa4ca1b7853
Sep 10 20:46:57.648208 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.648179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw" event={"ID":"80e3527b-f167-4892-81e0-139aed21aa9e","Type":"ContainerStarted","Data":"9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1"}
Sep 10 20:46:57.648364 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.648230    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw" event={"ID":"80e3527b-f167-4892-81e0-139aed21aa9e","Type":"ContainerStarted","Data":"14942445f7cb1a6f5fd5f5068f53017273d40cadfec1e35f12d51aa4ca1b7853"}
Sep 10 20:46:57.648364 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.648307    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.649930 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.649904    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"]
Sep 10 20:46:57.650991 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.650957    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.652221 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.652200    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"]
Sep 10 20:46:57.661747 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.661727    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"]
Sep 10 20:46:57.661885 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.661866    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" containerName="manager" containerID="cri-o://52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb" gracePeriod=2
Sep 10 20:46:57.665725 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.665681    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.672389 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.672334    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"]
Sep 10 20:46:57.679943 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.679923    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"]
Sep 10 20:46:57.680326 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.680314    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" containerName="manager"
Sep 10 20:46:57.680372 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.680329    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" containerName="manager"
Sep 10 20:46:57.680372 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.680338    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" containerName="manager"
Sep 10 20:46:57.680372 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.680343    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" containerName="manager"
Sep 10 20:46:57.680460 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.680408    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" containerName="manager"
Sep 10 20:46:57.680460 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.680418    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" containerName="manager"
Sep 10 20:46:57.683343 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.683327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:57.689350 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.689327    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.692025 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.691994    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:57.692860 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.692835    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"]
Sep 10 20:46:57.693043 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.693019    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" containerName="manager" containerID="cri-o://6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d" gracePeriod=2
Sep 10 20:46:57.715495 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.715366    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"]
Sep 10 20:46:57.717316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.717290    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"]
Sep 10 20:46:57.721946 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.721927    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"]
Sep 10 20:46:57.722078 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.722027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:57.726798 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.726778    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"]
Sep 10 20:46:57.727174 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.727158    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" containerName="manager"
Sep 10 20:46:57.727248 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.727177    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" containerName="manager"
Sep 10 20:46:57.727301 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.727293    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" containerName="manager"
Sep 10 20:46:57.730177 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.730157    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"
Sep 10 20:46:57.735256 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.735235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"]
Sep 10 20:46:57.744196 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.744175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"]
Sep 10 20:46:57.745966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.745939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df24263-be27-42f0-87cf-c3d2905dc420-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gp9s2\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:57.746068 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.746046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbjg\" (UniqueName: \"kubernetes.io/projected/3df24263-be27-42f0-87cf-c3d2905dc420-kube-api-access-4nbjg\") pod \"kuadrant-operator-controller-manager-5f685c496f-gp9s2\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:57.785401 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.785372    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wh2td\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:57.787792 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.787764    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.791667 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.791636    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:57.847425 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.847386    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8871bb0b-18ac-4210-add7-3a654a3d9ebe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdtc\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:57.847614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.847459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbs4d\" (UniqueName: \"kubernetes.io/projected/8871bb0b-18ac-4210-add7-3a654a3d9ebe-kube-api-access-sbs4d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdtc\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:57.847614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.847494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df24263-be27-42f0-87cf-c3d2905dc420-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gp9s2\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:57.847614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.847548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbjg\" (UniqueName: \"kubernetes.io/projected/3df24263-be27-42f0-87cf-c3d2905dc420-kube-api-access-4nbjg\") pod \"kuadrant-operator-controller-manager-5f685c496f-gp9s2\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:57.847984 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.847960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df24263-be27-42f0-87cf-c3d2905dc420-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gp9s2\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:57.848035 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.848019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xx6q\" (UniqueName: \"kubernetes.io/projected/96295e43-01f0-4f2b-a266-61b61e56285f-kube-api-access-2xx6q\") pod \"limitador-operator-controller-manager-74495c69f8-kftmv\" (UID: \"96295e43-01f0-4f2b-a266-61b61e56285f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"
Sep 10 20:46:57.862758 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.862729    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbjg\" (UniqueName: \"kubernetes.io/projected/3df24263-be27-42f0-87cf-c3d2905dc420-kube-api-access-4nbjg\") pod \"kuadrant-operator-controller-manager-5f685c496f-gp9s2\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:57.917885 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.917863    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:57.921006 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.920978    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wh2td\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:57.923361 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.923343    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:57.923872 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.923849    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.926865 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.926842    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:57.931505 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.931483    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wh2td\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:57.934041 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.934014    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:57.936530 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:57.936509    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-znrw7\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:57.948978 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.948959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8871bb0b-18ac-4210-add7-3a654a3d9ebe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdtc\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:57.949060 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.949003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbs4d\" (UniqueName: \"kubernetes.io/projected/8871bb0b-18ac-4210-add7-3a654a3d9ebe-kube-api-access-sbs4d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdtc\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:57.949112 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.949063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xx6q\" (UniqueName: \"kubernetes.io/projected/96295e43-01f0-4f2b-a266-61b61e56285f-kube-api-access-2xx6q\") pod \"limitador-operator-controller-manager-74495c69f8-kftmv\" (UID: \"96295e43-01f0-4f2b-a266-61b61e56285f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"
Sep 10 20:46:57.949355 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.949333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8871bb0b-18ac-4210-add7-3a654a3d9ebe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdtc\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:57.958247 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.958229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xx6q\" (UniqueName: \"kubernetes.io/projected/96295e43-01f0-4f2b-a266-61b61e56285f-kube-api-access-2xx6q\") pod \"limitador-operator-controller-manager-74495c69f8-kftmv\" (UID: \"96295e43-01f0-4f2b-a266-61b61e56285f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"
Sep 10 20:46:57.958480 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:57.958463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbs4d\" (UniqueName: \"kubernetes.io/projected/8871bb0b-18ac-4210-add7-3a654a3d9ebe-kube-api-access-sbs4d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngdtc\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:58.049784 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.049736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-extensions-socket-volume\") pod \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") "
Sep 10 20:46:58.049859 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.049836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj79z\" (UniqueName: \"kubernetes.io/projected/32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0-kube-api-access-hj79z\") pod \"32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0\" (UID: \"32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0\") "
Sep 10 20:46:58.049902 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.049892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg9dn\" (UniqueName: \"kubernetes.io/projected/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-kube-api-access-sg9dn\") pod \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\" (UID: \"4e83ea4d-d4fd-4415-8b5c-0ac968de9871\") "
Sep 10 20:46:58.050165 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.050132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4e83ea4d-d4fd-4415-8b5c-0ac968de9871" (UID: "4e83ea4d-d4fd-4415-8b5c-0ac968de9871"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:58.051843 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.051820    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0-kube-api-access-hj79z" (OuterVolumeSpecName: "kube-api-access-hj79z") pod "32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" (UID: "32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0"). InnerVolumeSpecName "kube-api-access-hj79z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:46:58.051915 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.051873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-kube-api-access-sg9dn" (OuterVolumeSpecName: "kube-api-access-sg9dn") pod "4e83ea4d-d4fd-4415-8b5c-0ac968de9871" (UID: "4e83ea4d-d4fd-4415-8b5c-0ac968de9871"). InnerVolumeSpecName "kube-api-access-sg9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:46:58.078028 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.078006    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:58.086113 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.086090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:58.092266 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.092242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"
Sep 10 20:46:58.151434 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.151182    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9dn\" (UniqueName: \"kubernetes.io/projected/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-kube-api-access-sg9dn\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:46:58.151434 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.151210    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e83ea4d-d4fd-4415-8b5c-0ac968de9871-extensions-socket-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:46:58.151434 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.151220    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj79z\" (UniqueName: \"kubernetes.io/projected/32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0-kube-api-access-hj79z\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:46:58.369614 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:58.369585    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df24263_be27_42f0_87cf_c3d2905dc420.slice/crio-40ff9568596a85e7254ebc0e18ba6f3ca18418df586afb343672b028095f9051 WatchSource:0}: Error finding container 40ff9568596a85e7254ebc0e18ba6f3ca18418df586afb343672b028095f9051: Status 404 returned error can't find the container with id 40ff9568596a85e7254ebc0e18ba6f3ca18418df586afb343672b028095f9051
Sep 10 20:46:58.371412 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.371385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"]
Sep 10 20:46:58.387791 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.387758    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"]
Sep 10 20:46:58.390325 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:58.390279    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8871bb0b_18ac_4210_add7_3a654a3d9ebe.slice/crio-7a13ebda34c6d94a07a31da4a9f66b7ac04ce5afe66ac423ba17aa5005591272 WatchSource:0}: Error finding container 7a13ebda34c6d94a07a31da4a9f66b7ac04ce5afe66ac423ba17aa5005591272: Status 404 returned error can't find the container with id 7a13ebda34c6d94a07a31da4a9f66b7ac04ce5afe66ac423ba17aa5005591272
Sep 10 20:46:58.390596 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.390579    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"]
Sep 10 20:46:58.392459 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:46:58.392425    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96295e43_01f0_4f2b_a266_61b61e56285f.slice/crio-808cb5bde95819893decc5d3e94efb303f78f895efab576e3d38ff6d467d4fa6 WatchSource:0}: Error finding container 808cb5bde95819893decc5d3e94efb303f78f895efab576e3d38ff6d467d4fa6: Status 404 returned error can't find the container with id 808cb5bde95819893decc5d3e94efb303f78f895efab576e3d38ff6d467d4fa6
Sep 10 20:46:58.593707 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.593665    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" path="/var/lib/kubelet/pods/32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0/volumes"
Sep 10 20:46:58.594057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.594043    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" path="/var/lib/kubelet/pods/4e83ea4d-d4fd-4415-8b5c-0ac968de9871/volumes"
Sep 10 20:46:58.656040 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.655958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc" event={"ID":"8871bb0b-18ac-4210-add7-3a654a3d9ebe","Type":"ContainerStarted","Data":"89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6"}
Sep 10 20:46:58.656040 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.656015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc" event={"ID":"8871bb0b-18ac-4210-add7-3a654a3d9ebe","Type":"ContainerStarted","Data":"7a13ebda34c6d94a07a31da4a9f66b7ac04ce5afe66ac423ba17aa5005591272"}
Sep 10 20:46:58.656197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.656047    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:46:58.657568 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.657544    2565 generic.go:356] "Generic (PLEG): container finished" podID="32acb98e-18d3-4cb2-b9eb-d6cfedc4b8f0" containerID="6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d" exitCode=0
Sep 10 20:46:58.657671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.657591    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wh2td"
Sep 10 20:46:58.657671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.657646    2565 scope.go:117] "RemoveContainer" containerID="6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d"
Sep 10 20:46:58.661427 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.661297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2" event={"ID":"3df24263-be27-42f0-87cf-c3d2905dc420","Type":"ContainerStarted","Data":"92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab"}
Sep 10 20:46:58.661427 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.661380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2" event={"ID":"3df24263-be27-42f0-87cf-c3d2905dc420","Type":"ContainerStarted","Data":"40ff9568596a85e7254ebc0e18ba6f3ca18418df586afb343672b028095f9051"}
Sep 10 20:46:58.661822 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.661631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:46:58.663461 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.663380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv" event={"ID":"96295e43-01f0-4f2b-a266-61b61e56285f","Type":"ContainerStarted","Data":"c8633f4c70c3aeb0ad0eb23b42921a7faf19f7215d877e6c2a9d588402aca6eb"}
Sep 10 20:46:58.663461 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.663429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv" event={"ID":"96295e43-01f0-4f2b-a266-61b61e56285f","Type":"ContainerStarted","Data":"808cb5bde95819893decc5d3e94efb303f78f895efab576e3d38ff6d467d4fa6"}
Sep 10 20:46:58.663695 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.663475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"
Sep 10 20:46:58.665156 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.665133    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e83ea4d-d4fd-4415-8b5c-0ac968de9871" containerID="52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb" exitCode=0
Sep 10 20:46:58.665263 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.665181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-znrw7"
Sep 10 20:46:58.665420 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:58.665406    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw" container="manager"
Sep 10 20:46:58.669122 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.669089    2565 scope.go:117] "RemoveContainer" containerID="6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d"
Sep 10 20:46:58.669368 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:58.669351    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d\": container with ID starting with 6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d not found: ID does not exist" containerID="6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d"
Sep 10 20:46:58.669447 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.669379    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d"} err="failed to get container status \"6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d\": rpc error: code = NotFound desc = could not find container \"6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d\": container with ID starting with 6617b92ed10cc95d0d9e9229a7a2538accf93ea01f8dd0940e30caf1d7b6b55d not found: ID does not exist"
Sep 10 20:46:58.669447 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.669403    2565 scope.go:117] "RemoveContainer" containerID="52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb"
Sep 10 20:46:58.678233 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.678217    2565 scope.go:117] "RemoveContainer" containerID="52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb"
Sep 10 20:46:58.678518 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:58.678490    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb\": container with ID starting with 52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb not found: ID does not exist" containerID="52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb"
Sep 10 20:46:58.678596 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.678522    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb"} err="failed to get container status \"52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb\": rpc error: code = NotFound desc = could not find container \"52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb\": container with ID starting with 52b809b9fac1049fb743bb4c39eb1a31325bccbad45e1986228423084e4cacfb not found: ID does not exist"
Sep 10 20:46:58.687804 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.687764    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc" podStartSLOduration=1.687751102 podStartE2EDuration="1.687751102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:58.685672213 +0000 UTC m=+654.596709497" watchObservedRunningTime="2026-09-10 20:46:58.687751102 +0000 UTC m=+654.598788385"
Sep 10 20:46:58.707267 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.707225    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2" podStartSLOduration=1.707204419 podStartE2EDuration="1.707204419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:58.706707554 +0000 UTC m=+654.617744832" watchObservedRunningTime="2026-09-10 20:46:58.707204419 +0000 UTC m=+654.618241702"
Sep 10 20:46:58.741450 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:58.741410    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv" podStartSLOduration=1.741398963 podStartE2EDuration="1.741398963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:46:58.73878823 +0000 UTC m=+654.649825514" watchObservedRunningTime="2026-09-10 20:46:58.741398963 +0000 UTC m=+654.652436247"
Sep 10 20:46:59.671743 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:59.671698    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" containerName="manager" containerID="cri-o://9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1" gracePeriod=2
Sep 10 20:46:59.674532 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:59.674507    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:59.903973 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:59.903948    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:59.909148 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:46:59.909119    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:46:59.968093 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:59.968039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80e3527b-f167-4892-81e0-139aed21aa9e-extensions-socket-volume\") pod \"80e3527b-f167-4892-81e0-139aed21aa9e\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") "
Sep 10 20:46:59.968093 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:59.968091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjjjv\" (UniqueName: \"kubernetes.io/projected/80e3527b-f167-4892-81e0-139aed21aa9e-kube-api-access-pjjjv\") pod \"80e3527b-f167-4892-81e0-139aed21aa9e\" (UID: \"80e3527b-f167-4892-81e0-139aed21aa9e\") "
Sep 10 20:46:59.968287 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:59.968266    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e3527b-f167-4892-81e0-139aed21aa9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "80e3527b-f167-4892-81e0-139aed21aa9e" (UID: "80e3527b-f167-4892-81e0-139aed21aa9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:59.970065 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:46:59.970037    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e3527b-f167-4892-81e0-139aed21aa9e-kube-api-access-pjjjv" (OuterVolumeSpecName: "kube-api-access-pjjjv") pod "80e3527b-f167-4892-81e0-139aed21aa9e" (UID: "80e3527b-f167-4892-81e0-139aed21aa9e"). InnerVolumeSpecName "kube-api-access-pjjjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:47:00.069380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.069351    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80e3527b-f167-4892-81e0-139aed21aa9e-extensions-socket-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:47:00.069380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.069375    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjjjv\" (UniqueName: \"kubernetes.io/projected/80e3527b-f167-4892-81e0-139aed21aa9e-kube-api-access-pjjjv\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:47:00.591237 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.591206    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" path="/var/lib/kubelet/pods/80e3527b-f167-4892-81e0-139aed21aa9e/volumes"
Sep 10 20:47:00.685782 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.685748    2565 generic.go:356] "Generic (PLEG): container finished" podID="80e3527b-f167-4892-81e0-139aed21aa9e" containerID="9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1" exitCode=2
Sep 10 20:47:00.686264 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.685802    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:47:00.686264 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.685837    2565 scope.go:117] "RemoveContainer" containerID="9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1"
Sep 10 20:47:00.690991 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:47:00.690969    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-k7blw\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="80e3527b-f167-4892-81e0-139aed21aa9e" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-k7blw"
Sep 10 20:47:00.694373 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.694361    2565 scope.go:117] "RemoveContainer" containerID="9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1"
Sep 10 20:47:00.694639 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:47:00.694620    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1\": container with ID starting with 9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1 not found: ID does not exist" containerID="9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1"
Sep 10 20:47:00.694701 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:00.694650    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1"} err="failed to get container status \"9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1\": rpc error: code = NotFound desc = could not find container \"9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1\": container with ID starting with 9c1b5182023be9a3a5f5b4a725918f0b49dc47085400ca53aefbd168eefca3d1 not found: ID does not exist"
Sep 10 20:47:09.676219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:09.676183    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:47:09.676665 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:09.676266    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:47:09.676665 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:09.676535    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kftmv"
Sep 10 20:47:09.773354 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:09.773324    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"]
Sep 10 20:47:09.773568 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:09.773542    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc" podUID="8871bb0b-18ac-4210-add7-3a654a3d9ebe" containerName="manager" containerID="cri-o://89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6" gracePeriod=10
Sep 10 20:47:10.013375 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.013345    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"]
Sep 10 20:47:10.035090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.035064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.045251 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.045227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"]
Sep 10 20:47:10.151885 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.151850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznxk\" (UniqueName: \"kubernetes.io/projected/b6e17378-c89c-42ec-925f-b79c9d34eec3-kube-api-access-hznxk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bsdzw\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.152041 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.151952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e17378-c89c-42ec-925f-b79c9d34eec3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bsdzw\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.252972 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.252896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e17378-c89c-42ec-925f-b79c9d34eec3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bsdzw\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.252972 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.252953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznxk\" (UniqueName: \"kubernetes.io/projected/b6e17378-c89c-42ec-925f-b79c9d34eec3-kube-api-access-hznxk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bsdzw\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.253329 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.253298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e17378-c89c-42ec-925f-b79c9d34eec3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bsdzw\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.268514 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.268485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznxk\" (UniqueName: \"kubernetes.io/projected/b6e17378-c89c-42ec-925f-b79c9d34eec3-kube-api-access-hznxk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bsdzw\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.359857 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.359830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:10.562449 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.562418    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:47:10.657054 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.657021    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbs4d\" (UniqueName: \"kubernetes.io/projected/8871bb0b-18ac-4210-add7-3a654a3d9ebe-kube-api-access-sbs4d\") pod \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") "
Sep 10 20:47:10.657228 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.657079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8871bb0b-18ac-4210-add7-3a654a3d9ebe-extensions-socket-volume\") pod \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\" (UID: \"8871bb0b-18ac-4210-add7-3a654a3d9ebe\") "
Sep 10 20:47:10.657518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.657495    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8871bb0b-18ac-4210-add7-3a654a3d9ebe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8871bb0b-18ac-4210-add7-3a654a3d9ebe" (UID: "8871bb0b-18ac-4210-add7-3a654a3d9ebe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:47:10.659789 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.659758    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8871bb0b-18ac-4210-add7-3a654a3d9ebe-kube-api-access-sbs4d" (OuterVolumeSpecName: "kube-api-access-sbs4d") pod "8871bb0b-18ac-4210-add7-3a654a3d9ebe" (UID: "8871bb0b-18ac-4210-add7-3a654a3d9ebe"). InnerVolumeSpecName "kube-api-access-sbs4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:47:10.661764 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.661744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"]
Sep 10 20:47:10.663757 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:47:10.663736    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e17378_c89c_42ec_925f_b79c9d34eec3.slice/crio-5884e7f9d9501761c77876e65cffdd814b96a861da266671d94ea02a3e5b2df2 WatchSource:0}: Error finding container 5884e7f9d9501761c77876e65cffdd814b96a861da266671d94ea02a3e5b2df2: Status 404 returned error can't find the container with id 5884e7f9d9501761c77876e65cffdd814b96a861da266671d94ea02a3e5b2df2
Sep 10 20:47:10.739900 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.739872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw" event={"ID":"b6e17378-c89c-42ec-925f-b79c9d34eec3","Type":"ContainerStarted","Data":"5884e7f9d9501761c77876e65cffdd814b96a861da266671d94ea02a3e5b2df2"}
Sep 10 20:47:10.741366 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.741343    2565 generic.go:356] "Generic (PLEG): container finished" podID="8871bb0b-18ac-4210-add7-3a654a3d9ebe" containerID="89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6" exitCode=0
Sep 10 20:47:10.741444 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.741391    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"
Sep 10 20:47:10.741444 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.741403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc" event={"ID":"8871bb0b-18ac-4210-add7-3a654a3d9ebe","Type":"ContainerDied","Data":"89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6"}
Sep 10 20:47:10.741444 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.741438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc" event={"ID":"8871bb0b-18ac-4210-add7-3a654a3d9ebe","Type":"ContainerDied","Data":"7a13ebda34c6d94a07a31da4a9f66b7ac04ce5afe66ac423ba17aa5005591272"}
Sep 10 20:47:10.741734 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.741459    2565 scope.go:117] "RemoveContainer" containerID="89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6"
Sep 10 20:47:10.749665 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.749640    2565 scope.go:117] "RemoveContainer" containerID="89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6"
Sep 10 20:47:10.749945 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:47:10.749927    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6\": container with ID starting with 89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6 not found: ID does not exist" containerID="89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6"
Sep 10 20:47:10.750000 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.749953    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6"} err="failed to get container status \"89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6\": rpc error: code = NotFound desc = could not find container \"89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6\": container with ID starting with 89dfb07badbb6df325de78db8b33332aa891e09af26e89d9e7088ebc70c0e4e6 not found: ID does not exist"
Sep 10 20:47:10.758603 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.758580    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbs4d\" (UniqueName: \"kubernetes.io/projected/8871bb0b-18ac-4210-add7-3a654a3d9ebe-kube-api-access-sbs4d\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:47:10.758718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.758606    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8871bb0b-18ac-4210-add7-3a654a3d9ebe-extensions-socket-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:47:10.766301 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.766278    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"]
Sep 10 20:47:10.775326 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:10.775306    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngdtc"]
Sep 10 20:47:11.748576 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:11.748537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw" event={"ID":"b6e17378-c89c-42ec-925f-b79c9d34eec3","Type":"ContainerStarted","Data":"c4fd859d33846cc42a7192e3d3318a9ca7f90b7066a6cc824a0a03780c6db9fa"}
Sep 10 20:47:11.749014 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:11.748666    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:11.778846 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:11.778808    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw" podStartSLOduration=2.778794164 podStartE2EDuration="2.778794164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:47:11.776989871 +0000 UTC m=+667.688027154" watchObservedRunningTime="2026-09-10 20:47:11.778794164 +0000 UTC m=+667.689831445"
Sep 10 20:47:12.591873 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:12.591837    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8871bb0b-18ac-4210-add7-3a654a3d9ebe" path="/var/lib/kubelet/pods/8871bb0b-18ac-4210-add7-3a654a3d9ebe/volumes"
Sep 10 20:47:22.757005 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:22.756976    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 20:47:22.820599 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:22.820567    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"]
Sep 10 20:47:22.820869 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:22.820826    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2" podUID="3df24263-be27-42f0-87cf-c3d2905dc420" containerName="manager" containerID="cri-o://92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab" gracePeriod=10
Sep 10 20:47:23.066575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.066551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:47:23.260883 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.260851    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nbjg\" (UniqueName: \"kubernetes.io/projected/3df24263-be27-42f0-87cf-c3d2905dc420-kube-api-access-4nbjg\") pod \"3df24263-be27-42f0-87cf-c3d2905dc420\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") "
Sep 10 20:47:23.261041 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.260898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df24263-be27-42f0-87cf-c3d2905dc420-extensions-socket-volume\") pod \"3df24263-be27-42f0-87cf-c3d2905dc420\" (UID: \"3df24263-be27-42f0-87cf-c3d2905dc420\") "
Sep 10 20:47:23.261276 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.261248    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df24263-be27-42f0-87cf-c3d2905dc420-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3df24263-be27-42f0-87cf-c3d2905dc420" (UID: "3df24263-be27-42f0-87cf-c3d2905dc420"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:47:23.262871 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.262853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df24263-be27-42f0-87cf-c3d2905dc420-kube-api-access-4nbjg" (OuterVolumeSpecName: "kube-api-access-4nbjg") pod "3df24263-be27-42f0-87cf-c3d2905dc420" (UID: "3df24263-be27-42f0-87cf-c3d2905dc420"). InnerVolumeSpecName "kube-api-access-4nbjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:47:23.362044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.362022    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nbjg\" (UniqueName: \"kubernetes.io/projected/3df24263-be27-42f0-87cf-c3d2905dc420-kube-api-access-4nbjg\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:47:23.362044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.362045    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df24263-be27-42f0-87cf-c3d2905dc420-extensions-socket-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:47:23.810315 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.810235    2565 generic.go:356] "Generic (PLEG): container finished" podID="3df24263-be27-42f0-87cf-c3d2905dc420" containerID="92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab" exitCode=0
Sep 10 20:47:23.810315 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.810278    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"
Sep 10 20:47:23.810790 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.810318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2" event={"ID":"3df24263-be27-42f0-87cf-c3d2905dc420","Type":"ContainerDied","Data":"92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab"}
Sep 10 20:47:23.810790 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.810364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2" event={"ID":"3df24263-be27-42f0-87cf-c3d2905dc420","Type":"ContainerDied","Data":"40ff9568596a85e7254ebc0e18ba6f3ca18418df586afb343672b028095f9051"}
Sep 10 20:47:23.810790 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.810385    2565 scope.go:117] "RemoveContainer" containerID="92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab"
Sep 10 20:47:23.821530 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.821509    2565 scope.go:117] "RemoveContainer" containerID="92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab"
Sep 10 20:47:23.822101 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:47:23.822072    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab\": container with ID starting with 92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab not found: ID does not exist" containerID="92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab"
Sep 10 20:47:23.822195 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.822103    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab"} err="failed to get container status \"92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab\": rpc error: code = NotFound desc = could not find container \"92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab\": container with ID starting with 92232b24e2663733e827f37f48bbde698c21d4579de3df413b36c3a71eccc9ab not found: ID does not exist"
Sep 10 20:47:23.839075 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.839054    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"]
Sep 10 20:47:23.842998 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:23.842979    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gp9s2"]
Sep 10 20:47:24.593840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:24.592172    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df24263-be27-42f0-87cf-c3d2905dc420" path="/var/lib/kubelet/pods/3df24263-be27-42f0-87cf-c3d2905dc420/volumes"
Sep 10 20:47:26.150666 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.150628    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"]
Sep 10 20:47:26.151234 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.151202    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df24263-be27-42f0-87cf-c3d2905dc420" containerName="manager"
Sep 10 20:47:26.151234 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.151225    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df24263-be27-42f0-87cf-c3d2905dc420" containerName="manager"
Sep 10 20:47:26.151369 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.151248    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8871bb0b-18ac-4210-add7-3a654a3d9ebe" containerName="manager"
Sep 10 20:47:26.151369 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.151257    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8871bb0b-18ac-4210-add7-3a654a3d9ebe" containerName="manager"
Sep 10 20:47:26.151369 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.151359    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df24263-be27-42f0-87cf-c3d2905dc420" containerName="manager"
Sep 10 20:47:26.151524 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.151377    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8871bb0b-18ac-4210-add7-3a654a3d9ebe" containerName="manager"
Sep 10 20:47:26.155215 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.155194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.158390 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.158371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 20:47:26.158490 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.158395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-k8hfd\""
Sep 10 20:47:26.166131 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.166111    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"]
Sep 10 20:47:26.184962 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.184931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62163f71-e629-4980-81da-76382d8e33ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185055 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.184980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62163f71-e629-4980-81da-76382d8e33ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185104 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.185051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.185100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185263 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.185143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8ln\" (UniqueName: \"kubernetes.io/projected/62163f71-e629-4980-81da-76382d8e33ff-kube-api-access-tt8ln\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185263 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.185186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62163f71-e629-4980-81da-76382d8e33ff-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185263 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.185237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185541 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.185354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.185623 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.185576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286438 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8ln\" (UniqueName: \"kubernetes.io/projected/62163f71-e629-4980-81da-76382d8e33ff-kube-api-access-tt8ln\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286604 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62163f71-e629-4980-81da-76382d8e33ff-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286650 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286730 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286791 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286791 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62163f71-e629-4980-81da-76382d8e33ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286891 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.286960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.286935    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62163f71-e629-4980-81da-76382d8e33ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.287079 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.287057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.287154 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.287131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.287211 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.287176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.287380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.287364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62163f71-e629-4980-81da-76382d8e33ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.288966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.288946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62163f71-e629-4980-81da-76382d8e33ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.289219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.289202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62163f71-e629-4980-81da-76382d8e33ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.295082 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.295058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62163f71-e629-4980-81da-76382d8e33ff-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.295179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.295084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8ln\" (UniqueName: \"kubernetes.io/projected/62163f71-e629-4980-81da-76382d8e33ff-kube-api-access-tt8ln\") pod \"maas-default-gateway-openshift-default-78bb78ff78-sspzh\" (UID: \"62163f71-e629-4980-81da-76382d8e33ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.465031 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.464971    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:26.739081 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.739053    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"]
Sep 10 20:47:26.740488 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:47:26.740459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62163f71_e629_4980_81da_76382d8e33ff.slice/crio-f5889c63145ea17937d6209b2c79a64ad9d6d7a8875bba24677f9b52e1d90ce3 WatchSource:0}: Error finding container f5889c63145ea17937d6209b2c79a64ad9d6d7a8875bba24677f9b52e1d90ce3: Status 404 returned error can't find the container with id f5889c63145ea17937d6209b2c79a64ad9d6d7a8875bba24677f9b52e1d90ce3
Sep 10 20:47:26.829820 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:26.829789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" event={"ID":"62163f71-e629-4980-81da-76382d8e33ff","Type":"ContainerStarted","Data":"f5889c63145ea17937d6209b2c79a64ad9d6d7a8875bba24677f9b52e1d90ce3"}
Sep 10 20:47:29.888001 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:29.887964    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:47:29.888224 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:29.888042    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:47:29.888224 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:29.888071    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:47:30.852223 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:30.852187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" event={"ID":"62163f71-e629-4980-81da-76382d8e33ff","Type":"ContainerStarted","Data":"15093078d97b39b57f078a09ab01e2ed0bb8ba67c90b8a51c1286b7a080baffe"}
Sep 10 20:47:30.877400 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:30.877355    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" podStartSLOduration=1.7322094890000002 podStartE2EDuration="4.877339197s" totalImagesPullingTime="3.145129708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:47:30.875192148 +0000 UTC m=+686.786229445" watchObservedRunningTime="2026-09-10 20:47:30.877339197 +0000 UTC m=+686.788376480"
Sep 10 20:47:31.465485 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:31.465451    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:31.469846 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:31.469821    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:31.857363 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:31.857337    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:31.858104 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:31.858084    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:47:40.787601 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:40.787563    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:47:40.791479 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:40.791457    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:40.794545 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:40.794526    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 20:47:40.804166 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:40.804138    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:47:40.836127 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:40.836103    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:47:40.912735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:40.912705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/380c068d-63eb-496d-a410-56a85b788331-config-file\") pod \"limitador-limitador-84994f7ffc-275xj\" (UID: \"380c068d-63eb-496d-a410-56a85b788331\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:40.912880 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:40.912746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nw7l\" (UniqueName: \"kubernetes.io/projected/380c068d-63eb-496d-a410-56a85b788331-kube-api-access-2nw7l\") pod \"limitador-limitador-84994f7ffc-275xj\" (UID: \"380c068d-63eb-496d-a410-56a85b788331\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:41.013400 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.013368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/380c068d-63eb-496d-a410-56a85b788331-config-file\") pod \"limitador-limitador-84994f7ffc-275xj\" (UID: \"380c068d-63eb-496d-a410-56a85b788331\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:41.013543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.013414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nw7l\" (UniqueName: \"kubernetes.io/projected/380c068d-63eb-496d-a410-56a85b788331-kube-api-access-2nw7l\") pod \"limitador-limitador-84994f7ffc-275xj\" (UID: \"380c068d-63eb-496d-a410-56a85b788331\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:41.014014 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.013992    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/380c068d-63eb-496d-a410-56a85b788331-config-file\") pod \"limitador-limitador-84994f7ffc-275xj\" (UID: \"380c068d-63eb-496d-a410-56a85b788331\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:41.022289 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.022260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nw7l\" (UniqueName: \"kubernetes.io/projected/380c068d-63eb-496d-a410-56a85b788331-kube-api-access-2nw7l\") pod \"limitador-limitador-84994f7ffc-275xj\" (UID: \"380c068d-63eb-496d-a410-56a85b788331\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:41.102389 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.102366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:41.108109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.108082    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jt62g"]
Sep 10 20:47:41.112724 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.112704    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g"
Sep 10 20:47:41.115646 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.115622    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zbpwx\""
Sep 10 20:47:41.123007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.122973    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jt62g"]
Sep 10 20:47:41.215530 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.215494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9l8\" (UniqueName: \"kubernetes.io/projected/8a506e24-069e-444f-9a47-6c139c86cb44-kube-api-access-tf9l8\") pod \"authorino-7bcfd8f9cd-jt62g\" (UID: \"8a506e24-069e-444f-9a47-6c139c86cb44\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g"
Sep 10 20:47:41.316908 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.316885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9l8\" (UniqueName: \"kubernetes.io/projected/8a506e24-069e-444f-9a47-6c139c86cb44-kube-api-access-tf9l8\") pod \"authorino-7bcfd8f9cd-jt62g\" (UID: \"8a506e24-069e-444f-9a47-6c139c86cb44\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g"
Sep 10 20:47:41.326663 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.326635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9l8\" (UniqueName: \"kubernetes.io/projected/8a506e24-069e-444f-9a47-6c139c86cb44-kube-api-access-tf9l8\") pod \"authorino-7bcfd8f9cd-jt62g\" (UID: \"8a506e24-069e-444f-9a47-6c139c86cb44\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g"
Sep 10 20:47:41.377437 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.377415    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:47:41.379612 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:47:41.379587    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380c068d_63eb_496d_a410_56a85b788331.slice/crio-072eeac02ad573b76ead99c91655f3fec68f216e8dafa385cad173b56ccb44a8 WatchSource:0}: Error finding container 072eeac02ad573b76ead99c91655f3fec68f216e8dafa385cad173b56ccb44a8: Status 404 returned error can't find the container with id 072eeac02ad573b76ead99c91655f3fec68f216e8dafa385cad173b56ccb44a8
Sep 10 20:47:41.437581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.437556    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g"
Sep 10 20:47:41.709967 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.709939    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jt62g"]
Sep 10 20:47:41.711086 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:47:41.711059    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a506e24_069e_444f_9a47_6c139c86cb44.slice/crio-c26728fc5b23f77147e0532beff5e64686968d30a5d167bc094f79eaae0cb2f8 WatchSource:0}: Error finding container c26728fc5b23f77147e0532beff5e64686968d30a5d167bc094f79eaae0cb2f8: Status 404 returned error can't find the container with id c26728fc5b23f77147e0532beff5e64686968d30a5d167bc094f79eaae0cb2f8
Sep 10 20:47:41.913253 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.913215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g" event={"ID":"8a506e24-069e-444f-9a47-6c139c86cb44","Type":"ContainerStarted","Data":"c26728fc5b23f77147e0532beff5e64686968d30a5d167bc094f79eaae0cb2f8"}
Sep 10 20:47:41.914455 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:41.914428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj" event={"ID":"380c068d-63eb-496d-a410-56a85b788331","Type":"ContainerStarted","Data":"072eeac02ad573b76ead99c91655f3fec68f216e8dafa385cad173b56ccb44a8"}
Sep 10 20:47:42.923633 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:42.923597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj" event={"ID":"380c068d-63eb-496d-a410-56a85b788331","Type":"ContainerStarted","Data":"ac94edeccdbf15bbaf47ec747fda802381507a9980f12dbc11600bcc1fb2e214"}
Sep 10 20:47:42.944625 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:42.944562    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj" podStartSLOduration=1.767240122 podStartE2EDuration="2.944543634s" totalImagesPullingTime="1.177303512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:47:42.942821921 +0000 UTC m=+698.853859206" watchObservedRunningTime="2026-09-10 20:47:42.944543634 +0000 UTC m=+698.855580919"
Sep 10 20:47:43.930296 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:43.930257    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g" event={"ID":"8a506e24-069e-444f-9a47-6c139c86cb44","Type":"ContainerStarted","Data":"1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742"}
Sep 10 20:47:43.930771 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:43.930349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:47:43.950247 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:43.950203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g" podStartSLOduration=1.142768356 podStartE2EDuration="2.950190317s" totalImagesPullingTime="1.807421961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:47:43.948215098 +0000 UTC m=+699.859252382" watchObservedRunningTime="2026-09-10 20:47:43.950190317 +0000 UTC m=+699.861227600"
Sep 10 20:47:44.532228 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:44.532191    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jt62g"]
Sep 10 20:47:45.940586 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:45.940544    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g" podUID="8a506e24-069e-444f-9a47-6c139c86cb44" containerName="authorino" containerID="cri-o://1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742" gracePeriod=30
Sep 10 20:47:46.184557 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.184533    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g"
Sep 10 20:47:46.260694 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.260626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf9l8\" (UniqueName: \"kubernetes.io/projected/8a506e24-069e-444f-9a47-6c139c86cb44-kube-api-access-tf9l8\") pod \"8a506e24-069e-444f-9a47-6c139c86cb44\" (UID: \"8a506e24-069e-444f-9a47-6c139c86cb44\") "
Sep 10 20:47:46.262574 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.262547    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a506e24-069e-444f-9a47-6c139c86cb44-kube-api-access-tf9l8" (OuterVolumeSpecName: "kube-api-access-tf9l8") pod "8a506e24-069e-444f-9a47-6c139c86cb44" (UID: "8a506e24-069e-444f-9a47-6c139c86cb44"). InnerVolumeSpecName "kube-api-access-tf9l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:47:46.361532 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.361505    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9l8\" (UniqueName: \"kubernetes.io/projected/8a506e24-069e-444f-9a47-6c139c86cb44-kube-api-access-tf9l8\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:47:46.946491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.946396    2565 generic.go:356] "Generic (PLEG): container finished" podID="8a506e24-069e-444f-9a47-6c139c86cb44" containerID="1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742" exitCode=0
Sep 10 20:47:46.946491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.946442    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g"
Sep 10 20:47:46.946491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.946455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g" event={"ID":"8a506e24-069e-444f-9a47-6c139c86cb44","Type":"ContainerDied","Data":"1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742"}
Sep 10 20:47:46.947003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.946500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jt62g" event={"ID":"8a506e24-069e-444f-9a47-6c139c86cb44","Type":"ContainerDied","Data":"c26728fc5b23f77147e0532beff5e64686968d30a5d167bc094f79eaae0cb2f8"}
Sep 10 20:47:46.947003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.946522    2565 scope.go:117] "RemoveContainer" containerID="1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742"
Sep 10 20:47:46.955238 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.955220    2565 scope.go:117] "RemoveContainer" containerID="1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742"
Sep 10 20:47:46.955503 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:47:46.955482    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742\": container with ID starting with 1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742 not found: ID does not exist" containerID="1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742"
Sep 10 20:47:46.955547 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.955512    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742"} err="failed to get container status \"1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742\": rpc error: code = NotFound desc = could not find container \"1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742\": container with ID starting with 1084fc5d8cba310a7691ee67a9dc5adc734615e90c99f7a1c796a6cc66014742 not found: ID does not exist"
Sep 10 20:47:46.964727 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.964706    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jt62g"]
Sep 10 20:47:46.966552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:46.966532    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jt62g"]
Sep 10 20:47:48.591134 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:48.591105    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a506e24-069e-444f-9a47-6c139c86cb44" path="/var/lib/kubelet/pods/8a506e24-069e-444f-9a47-6c139c86cb44/volumes"
Sep 10 20:47:54.935912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:47:54.935883    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-275xj"
Sep 10 20:48:10.900060 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:10.900026    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-5pcjk"]
Sep 10 20:48:10.900608 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:10.900583    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a506e24-069e-444f-9a47-6c139c86cb44" containerName="authorino"
Sep 10 20:48:10.900608 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:10.900607    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a506e24-069e-444f-9a47-6c139c86cb44" containerName="authorino"
Sep 10 20:48:10.900810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:10.900722    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a506e24-069e-444f-9a47-6c139c86cb44" containerName="authorino"
Sep 10 20:48:10.909937 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:10.909916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5pcjk"
Sep 10 20:48:10.912306 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:10.912282    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-5pcjk"]
Sep 10 20:48:10.913769 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:10.913747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zbpwx\""
Sep 10 20:48:11.064531 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:11.064499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfct\" (UniqueName: \"kubernetes.io/projected/5bfa0f5d-5b74-49a5-936f-e7dee3a1676e-kube-api-access-hzfct\") pod \"authorino-557f876f69-5pcjk\" (UID: \"5bfa0f5d-5b74-49a5-936f-e7dee3a1676e\") " pod="kuadrant-system/authorino-557f876f69-5pcjk"
Sep 10 20:48:11.140051 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:11.140020    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5pcjk"]
Sep 10 20:48:11.140274 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:48:11.140255    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hzfct], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-5pcjk" podUID="5bfa0f5d-5b74-49a5-936f-e7dee3a1676e"
Sep 10 20:48:11.165420 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:11.165342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfct\" (UniqueName: \"kubernetes.io/projected/5bfa0f5d-5b74-49a5-936f-e7dee3a1676e-kube-api-access-hzfct\") pod \"authorino-557f876f69-5pcjk\" (UID: \"5bfa0f5d-5b74-49a5-936f-e7dee3a1676e\") " pod="kuadrant-system/authorino-557f876f69-5pcjk"
Sep 10 20:48:11.175049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:11.175019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfct\" (UniqueName: \"kubernetes.io/projected/5bfa0f5d-5b74-49a5-936f-e7dee3a1676e-kube-api-access-hzfct\") pod \"authorino-557f876f69-5pcjk\" (UID: \"5bfa0f5d-5b74-49a5-936f-e7dee3a1676e\") " pod="kuadrant-system/authorino-557f876f69-5pcjk"
Sep 10 20:48:12.073933 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.073899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5pcjk"
Sep 10 20:48:12.081924 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.081901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5pcjk"
Sep 10 20:48:12.276151 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.276126    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzfct\" (UniqueName: \"kubernetes.io/projected/5bfa0f5d-5b74-49a5-936f-e7dee3a1676e-kube-api-access-hzfct\") pod \"5bfa0f5d-5b74-49a5-936f-e7dee3a1676e\" (UID: \"5bfa0f5d-5b74-49a5-936f-e7dee3a1676e\") "
Sep 10 20:48:12.278209 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.278177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa0f5d-5b74-49a5-936f-e7dee3a1676e-kube-api-access-hzfct" (OuterVolumeSpecName: "kube-api-access-hzfct") pod "5bfa0f5d-5b74-49a5-936f-e7dee3a1676e" (UID: "5bfa0f5d-5b74-49a5-936f-e7dee3a1676e"). InnerVolumeSpecName "kube-api-access-hzfct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:48:12.376754 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.376729    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfct\" (UniqueName: \"kubernetes.io/projected/5bfa0f5d-5b74-49a5-936f-e7dee3a1676e-kube-api-access-hzfct\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:48:12.759339 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.759270    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-569b688d9d-wjvj6"]
Sep 10 20:48:12.763385 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.763364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:12.766428 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.766408    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 20:48:12.770876 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.770801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-569b688d9d-wjvj6"]
Sep 10 20:48:12.881491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.881460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2455106a-dd1c-419d-8eda-0c0314aaaa55-tls-cert\") pod \"authorino-569b688d9d-wjvj6\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") " pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:12.881663 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.881602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pxt\" (UniqueName: \"kubernetes.io/projected/2455106a-dd1c-419d-8eda-0c0314aaaa55-kube-api-access-z2pxt\") pod \"authorino-569b688d9d-wjvj6\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") " pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:12.982827 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.982800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2455106a-dd1c-419d-8eda-0c0314aaaa55-tls-cert\") pod \"authorino-569b688d9d-wjvj6\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") " pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:12.982960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.982858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pxt\" (UniqueName: \"kubernetes.io/projected/2455106a-dd1c-419d-8eda-0c0314aaaa55-kube-api-access-z2pxt\") pod \"authorino-569b688d9d-wjvj6\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") " pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:12.985146 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.985127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2455106a-dd1c-419d-8eda-0c0314aaaa55-tls-cert\") pod \"authorino-569b688d9d-wjvj6\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") " pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:12.991722 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:12.991672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pxt\" (UniqueName: \"kubernetes.io/projected/2455106a-dd1c-419d-8eda-0c0314aaaa55-kube-api-access-z2pxt\") pod \"authorino-569b688d9d-wjvj6\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") " pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:13.074296 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:13.074207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:48:13.078567 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:13.078543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5pcjk"
Sep 10 20:48:13.115468 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:13.115442    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5pcjk"]
Sep 10 20:48:13.117392 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:13.117367    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-5pcjk"]
Sep 10 20:48:13.350741 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:13.350718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-569b688d9d-wjvj6"]
Sep 10 20:48:13.352623 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:48:13.352598    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2455106a_dd1c_419d_8eda_0c0314aaaa55.slice/crio-5ae1967794b9f9728c5bf1aa75c1435cd25bc19ae99e1dc3c866d5ce8e807078 WatchSource:0}: Error finding container 5ae1967794b9f9728c5bf1aa75c1435cd25bc19ae99e1dc3c866d5ce8e807078: Status 404 returned error can't find the container with id 5ae1967794b9f9728c5bf1aa75c1435cd25bc19ae99e1dc3c866d5ce8e807078
Sep 10 20:48:14.085230 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:14.085191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-569b688d9d-wjvj6" event={"ID":"2455106a-dd1c-419d-8eda-0c0314aaaa55","Type":"ContainerStarted","Data":"c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127"}
Sep 10 20:48:14.085615 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:14.085240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-569b688d9d-wjvj6" event={"ID":"2455106a-dd1c-419d-8eda-0c0314aaaa55","Type":"ContainerStarted","Data":"5ae1967794b9f9728c5bf1aa75c1435cd25bc19ae99e1dc3c866d5ce8e807078"}
Sep 10 20:48:14.105044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:14.104955    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-569b688d9d-wjvj6" podStartSLOduration=1.604905268 podStartE2EDuration="2.104938296s" totalImagesPullingTime="500.033028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:48:14.104401416 +0000 UTC m=+730.015438701" watchObservedRunningTime="2026-09-10 20:48:14.104938296 +0000 UTC m=+730.015975580"
Sep 10 20:48:14.590638 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:14.590612    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bfa0f5d-5b74-49a5-936f-e7dee3a1676e" path="/var/lib/kubelet/pods/5bfa0f5d-5b74-49a5-936f-e7dee3a1676e/volumes"
Sep 10 20:48:55.678738 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.678700    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"]
Sep 10 20:48:55.683007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.682983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:55.686459 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.686432    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 20:48:55.686606 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.686499    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pd4w9\""
Sep 10 20:48:55.694446 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.694422    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"]
Sep 10 20:48:55.706457 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.706424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:48:55.842945 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.842905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbpx\" (UniqueName: \"kubernetes.io/projected/278ec5f3-c99d-47bb-99d1-0185806fe1e0-kube-api-access-sqbpx\") pod \"payload-processing-5b54b7fbb7-lq8k2\" (UID: \"278ec5f3-c99d-47bb-99d1-0185806fe1e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:55.843128 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.843042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278ec5f3-c99d-47bb-99d1-0185806fe1e0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lq8k2\" (UID: \"278ec5f3-c99d-47bb-99d1-0185806fe1e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:55.943960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.943876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278ec5f3-c99d-47bb-99d1-0185806fe1e0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lq8k2\" (UID: \"278ec5f3-c99d-47bb-99d1-0185806fe1e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:55.944099 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.944001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbpx\" (UniqueName: \"kubernetes.io/projected/278ec5f3-c99d-47bb-99d1-0185806fe1e0-kube-api-access-sqbpx\") pod \"payload-processing-5b54b7fbb7-lq8k2\" (UID: \"278ec5f3-c99d-47bb-99d1-0185806fe1e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:55.944586 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.944566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278ec5f3-c99d-47bb-99d1-0185806fe1e0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lq8k2\" (UID: \"278ec5f3-c99d-47bb-99d1-0185806fe1e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:55.957104 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.957055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbpx\" (UniqueName: \"kubernetes.io/projected/278ec5f3-c99d-47bb-99d1-0185806fe1e0-kube-api-access-sqbpx\") pod \"payload-processing-5b54b7fbb7-lq8k2\" (UID: \"278ec5f3-c99d-47bb-99d1-0185806fe1e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:55.997595 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:55.997562    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:48:56.277251 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:56.277224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"]
Sep 10 20:48:56.278393 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:48:56.278369    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278ec5f3_c99d_47bb_99d1_0185806fe1e0.slice/crio-278f09094b0fc784d59a901f39de86c906809a5e7b99d8c14b207aadff9839a1 WatchSource:0}: Error finding container 278f09094b0fc784d59a901f39de86c906809a5e7b99d8c14b207aadff9839a1: Status 404 returned error can't find the container with id 278f09094b0fc784d59a901f39de86c906809a5e7b99d8c14b207aadff9839a1
Sep 10 20:48:56.280240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:56.280222    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:48:56.301647 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:56.301618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2" event={"ID":"278ec5f3-c99d-47bb-99d1-0185806fe1e0","Type":"ContainerStarted","Data":"278f09094b0fc784d59a901f39de86c906809a5e7b99d8c14b207aadff9839a1"}
Sep 10 20:48:59.321614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:59.321577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2" event={"ID":"278ec5f3-c99d-47bb-99d1-0185806fe1e0","Type":"ContainerStarted","Data":"009f54a97cb90b6f1bdbb73dbe7a20356de4b8f339e9f54da37891be6f1af340"}
Sep 10 20:48:59.322079 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:48:59.321669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:49:02.899923 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:02.899876    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-sspzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 20:49:02.900318 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:02.899979    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" podUID="62163f71-e629-4980-81da-76382d8e33ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:49:05.365265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:05.365234    2565 generic.go:356] "Generic (PLEG): container finished" podID="62163f71-e629-4980-81da-76382d8e33ff" containerID="15093078d97b39b57f078a09ab01e2ed0bb8ba67c90b8a51c1286b7a080baffe" exitCode=0
Sep 10 20:49:05.365668 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:05.365311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" event={"ID":"62163f71-e629-4980-81da-76382d8e33ff","Type":"ContainerDied","Data":"15093078d97b39b57f078a09ab01e2ed0bb8ba67c90b8a51c1286b7a080baffe"}
Sep 10 20:49:05.365734 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:05.365721    2565 scope.go:117] "RemoveContainer" containerID="15093078d97b39b57f078a09ab01e2ed0bb8ba67c90b8a51c1286b7a080baffe"
Sep 10 20:49:05.366791 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:05.366758    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:49:05.366865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:05.366830    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:49:05.366865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:05.366860    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:49:05.390584 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:05.390541    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2" podStartSLOduration=8.269440025 podStartE2EDuration="10.390525883s" totalImagesPullingTime="2.121085858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:48:59.348395045 +0000 UTC m=+775.259432333" watchObservedRunningTime="2026-09-10 20:49:05.390525883 +0000 UTC m=+781.301563160"
Sep 10 20:49:06.372763 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:06.372726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" event={"ID":"62163f71-e629-4980-81da-76382d8e33ff","Type":"ContainerStarted","Data":"e8dedbe9c03f275da3affec764846cecb26c3e6e48f0835d3933561c7d636c55"}
Sep 10 20:49:06.465874 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:06.465835    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:49:06.466054 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:06.466035    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-sspzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 20:49:06.466110 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:06.466090    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" podUID="62163f71-e629-4980-81da-76382d8e33ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 20:49:07.465611 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:07.465573    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-sspzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 20:49:07.466000 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:07.465627    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" podUID="62163f71-e629-4980-81da-76382d8e33ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 20:49:08.465781 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:08.465743    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-sspzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 20:49:08.466159 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:08.465799    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" podUID="62163f71-e629-4980-81da-76382d8e33ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 20:49:09.466132 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:09.466098    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-sspzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 20:49:09.466516 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:09.466151    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" podUID="62163f71-e629-4980-81da-76382d8e33ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 20:49:10.329636 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:10.329599    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lq8k2"
Sep 10 20:49:10.465865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:10.465827    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-sspzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 20:49:10.466037 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:10.465882    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh" podUID="62163f71-e629-4980-81da-76382d8e33ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 20:49:11.468673 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:11.468645    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:49:11.469142 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:11.468963    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:49:11.469659 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:11.469640    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-sspzh"
Sep 10 20:49:25.502982 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:25.502944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:49:33.009000 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:33.008922    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:49:49.794171 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.794140    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"]
Sep 10 20:49:49.798630 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.798610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:49.803925 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.803398    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kzwmk\""
Sep 10 20:49:49.803925 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.803481    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 20:49:49.803925 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.803401    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 20:49:49.803925 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.803765    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 20:49:49.806236 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.806211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"]
Sep 10 20:49:49.912168 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.912136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:49.912168 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.912166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:49.912380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.912242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf8d113-d98f-4062-96b2-2f26d2a45f62-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:49.912380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.912262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8x5t\" (UniqueName: \"kubernetes.io/projected/cdf8d113-d98f-4062-96b2-2f26d2a45f62-kube-api-access-c8x5t\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:49.912380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.912293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:49.912380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:49.912319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.013510 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.013460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf8d113-d98f-4062-96b2-2f26d2a45f62-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.013510 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.013517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8x5t\" (UniqueName: \"kubernetes.io/projected/cdf8d113-d98f-4062-96b2-2f26d2a45f62-kube-api-access-c8x5t\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.013810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.013544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.013810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.013562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.013810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.013644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.013810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.013665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.014062 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.014036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.014120 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.014074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.014173 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.014133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.015911 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.015889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d113-d98f-4062-96b2-2f26d2a45f62-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.016050 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.016033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf8d113-d98f-4062-96b2-2f26d2a45f62-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.023495 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.023475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8x5t\" (UniqueName: \"kubernetes.io/projected/cdf8d113-d98f-4062-96b2-2f26d2a45f62-kube-api-access-c8x5t\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7\" (UID: \"cdf8d113-d98f-4062-96b2-2f26d2a45f62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.111581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.111550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:49:50.383072 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.382998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"]
Sep 10 20:49:50.386304 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:49:50.386275    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf8d113_d98f_4062_96b2_2f26d2a45f62.slice/crio-1f7f7ae70ad344c4b05c7ecedbf883ad1d6a9cb7bc14aa58803fbbf2c2591a95 WatchSource:0}: Error finding container 1f7f7ae70ad344c4b05c7ecedbf883ad1d6a9cb7bc14aa58803fbbf2c2591a95: Status 404 returned error can't find the container with id 1f7f7ae70ad344c4b05c7ecedbf883ad1d6a9cb7bc14aa58803fbbf2c2591a95
Sep 10 20:49:50.617372 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.617341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7" event={"ID":"cdf8d113-d98f-4062-96b2-2f26d2a45f62","Type":"ContainerStarted","Data":"1f7f7ae70ad344c4b05c7ecedbf883ad1d6a9cb7bc14aa58803fbbf2c2591a95"}
Sep 10 20:49:50.717016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:50.716932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:49:51.625784 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:51.625740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7" event={"ID":"cdf8d113-d98f-4062-96b2-2f26d2a45f62","Type":"ContainerStarted","Data":"83ac2e5cd766e5ab0658dc560ebb0d4f926b5e449e96c65202d247c5047bf773"}
Sep 10 20:49:51.652608 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:49:51.652550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7" podStartSLOduration=1.573218488 podStartE2EDuration="2.652536439s" totalImagesPullingTime="1.079317951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:49:51.651172326 +0000 UTC m=+827.562209633" watchObservedRunningTime="2026-09-10 20:49:51.652536439 +0000 UTC m=+827.563573723"
Sep 10 20:50:00.111814 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:00.111775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:50:00.112267 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:00.111837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:50:00.124175 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:00.124149    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:50:00.682305 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:00.682277    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7"
Sep 10 20:50:06.999463 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:06.999426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:50:12.000149 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:12.000116    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:50:19.691309 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:19.691271    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:50:48.699390 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:50:48.699354    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:51:04.579003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:04.578927    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:51:04.580720 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:04.580695    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:51:04.582724 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:04.582679    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:51:04.584702 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:04.584669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:51:51.763175 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:51.763142    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-569b688d9d-wjvj6"]
Sep 10 20:51:51.763602 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:51.763318    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-569b688d9d-wjvj6" podUID="2455106a-dd1c-419d-8eda-0c0314aaaa55" containerName="authorino" containerID="cri-o://c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127" gracePeriod=30
Sep 10 20:51:52.020662 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.020605    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:51:52.055048 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.055019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2455106a-dd1c-419d-8eda-0c0314aaaa55-tls-cert\") pod \"2455106a-dd1c-419d-8eda-0c0314aaaa55\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") "
Sep 10 20:51:52.055191 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.055054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2pxt\" (UniqueName: \"kubernetes.io/projected/2455106a-dd1c-419d-8eda-0c0314aaaa55-kube-api-access-z2pxt\") pod \"2455106a-dd1c-419d-8eda-0c0314aaaa55\" (UID: \"2455106a-dd1c-419d-8eda-0c0314aaaa55\") "
Sep 10 20:51:52.057236 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.057201    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2455106a-dd1c-419d-8eda-0c0314aaaa55-kube-api-access-z2pxt" (OuterVolumeSpecName: "kube-api-access-z2pxt") pod "2455106a-dd1c-419d-8eda-0c0314aaaa55" (UID: "2455106a-dd1c-419d-8eda-0c0314aaaa55"). InnerVolumeSpecName "kube-api-access-z2pxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:51:52.065210 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.065182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2455106a-dd1c-419d-8eda-0c0314aaaa55-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2455106a-dd1c-419d-8eda-0c0314aaaa55" (UID: "2455106a-dd1c-419d-8eda-0c0314aaaa55"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:51:52.156400 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.156374    2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2455106a-dd1c-419d-8eda-0c0314aaaa55-tls-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:51:52.156400 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.156398    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2pxt\" (UniqueName: \"kubernetes.io/projected/2455106a-dd1c-419d-8eda-0c0314aaaa55-kube-api-access-z2pxt\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:51:52.278362 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.278291    2565 generic.go:356] "Generic (PLEG): container finished" podID="2455106a-dd1c-419d-8eda-0c0314aaaa55" containerID="c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127" exitCode=0
Sep 10 20:51:52.278362 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.278333    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569b688d9d-wjvj6"
Sep 10 20:51:52.278524 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.278336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-569b688d9d-wjvj6" event={"ID":"2455106a-dd1c-419d-8eda-0c0314aaaa55","Type":"ContainerDied","Data":"c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127"}
Sep 10 20:51:52.278524 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.278457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-569b688d9d-wjvj6" event={"ID":"2455106a-dd1c-419d-8eda-0c0314aaaa55","Type":"ContainerDied","Data":"5ae1967794b9f9728c5bf1aa75c1435cd25bc19ae99e1dc3c866d5ce8e807078"}
Sep 10 20:51:52.278524 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.278479    2565 scope.go:117] "RemoveContainer" containerID="c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127"
Sep 10 20:51:52.288634 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.288617    2565 scope.go:117] "RemoveContainer" containerID="c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127"
Sep 10 20:51:52.288959 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:51:52.288938    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127\": container with ID starting with c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127 not found: ID does not exist" containerID="c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127"
Sep 10 20:51:52.289031 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.288971    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127"} err="failed to get container status \"c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127\": rpc error: code = NotFound desc = could not find container \"c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127\": container with ID starting with c0f71c7391e029d82a6f3838a7a25a624eee28e63ff27283d67b417fd76f5127 not found: ID does not exist"
Sep 10 20:51:52.308032 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.308006    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-569b688d9d-wjvj6"]
Sep 10 20:51:52.313994 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.313973    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-569b688d9d-wjvj6"]
Sep 10 20:51:52.590283 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:51:52.590254    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2455106a-dd1c-419d-8eda-0c0314aaaa55" path="/var/lib/kubelet/pods/2455106a-dd1c-419d-8eda-0c0314aaaa55/volumes"
Sep 10 20:52:18.434523 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.434484    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8029-50f346"]
Sep 10 20:52:18.435179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.435156    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2455106a-dd1c-419d-8eda-0c0314aaaa55" containerName="authorino"
Sep 10 20:52:18.435323 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.435180    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2455106a-dd1c-419d-8eda-0c0314aaaa55" containerName="authorino"
Sep 10 20:52:18.435385 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.435321    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2455106a-dd1c-419d-8eda-0c0314aaaa55" containerName="authorino"
Sep 10 20:52:18.439782 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.439759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:18.443137 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.443009    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-64jfg\""
Sep 10 20:52:18.446302 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.446276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8029-50f346"]
Sep 10 20:52:18.574837 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.574808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd4xq\" (UniqueName: \"kubernetes.io/projected/c39ef6ce-8e33-44c0-894a-e1e30ef2ff84-kube-api-access-hd4xq\") pod \"test-curl-8029-50f346\" (UID: \"c39ef6ce-8e33-44c0-894a-e1e30ef2ff84\") " pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:18.675626 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.675593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd4xq\" (UniqueName: \"kubernetes.io/projected/c39ef6ce-8e33-44c0-894a-e1e30ef2ff84-kube-api-access-hd4xq\") pod \"test-curl-8029-50f346\" (UID: \"c39ef6ce-8e33-44c0-894a-e1e30ef2ff84\") " pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:18.684322 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.684299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd4xq\" (UniqueName: \"kubernetes.io/projected/c39ef6ce-8e33-44c0-894a-e1e30ef2ff84-kube-api-access-hd4xq\") pod \"test-curl-8029-50f346\" (UID: \"c39ef6ce-8e33-44c0-894a-e1e30ef2ff84\") " pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:18.750351 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:18.750290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:19.020297 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:19.020271    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8029-50f346"]
Sep 10 20:52:19.022121 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:19.022093    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc39ef6ce_8e33_44c0_894a_e1e30ef2ff84.slice/crio-8a031a9c7656439c9e8a03e426fb02e3aaa3da0fd1c37ba3f31b548dffe1587d WatchSource:0}: Error finding container 8a031a9c7656439c9e8a03e426fb02e3aaa3da0fd1c37ba3f31b548dffe1587d: Status 404 returned error can't find the container with id 8a031a9c7656439c9e8a03e426fb02e3aaa3da0fd1c37ba3f31b548dffe1587d
Sep 10 20:52:19.421653 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:19.421615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8029-50f346" event={"ID":"c39ef6ce-8e33-44c0-894a-e1e30ef2ff84","Type":"ContainerStarted","Data":"8a031a9c7656439c9e8a03e426fb02e3aaa3da0fd1c37ba3f31b548dffe1587d"}
Sep 10 20:52:20.230481 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:20.230446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:52:21.762451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:21.762429    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"]
Sep 10 20:52:21.766331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:21.766311    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:21.769307 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:21.769289    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-80aa25-dockercfg-rnjfh\""
Sep 10 20:52:21.769846 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:21.769609    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-80aa25\""
Sep 10 20:52:21.778303 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:21.778282    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"]
Sep 10 20:52:21.907909 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:21.907875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3c7d77-b4a0-4f87-9856-1e1a618db392-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80aa25-d8499f755-557zk\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:21.908075 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:21.907934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzs9n\" (UniqueName: \"kubernetes.io/projected/db3c7d77-b4a0-4f87-9856-1e1a618db392-kube-api-access-mzs9n\") pod \"payload-processing-e2e-inf-a-80aa25-d8499f755-557zk\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:22.008583 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.008511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3c7d77-b4a0-4f87-9856-1e1a618db392-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80aa25-d8499f755-557zk\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:22.008583 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.008554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzs9n\" (UniqueName: \"kubernetes.io/projected/db3c7d77-b4a0-4f87-9856-1e1a618db392-kube-api-access-mzs9n\") pod \"payload-processing-e2e-inf-a-80aa25-d8499f755-557zk\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:22.009327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.009302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3c7d77-b4a0-4f87-9856-1e1a618db392-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80aa25-d8499f755-557zk\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:22.017853 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.017826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzs9n\" (UniqueName: \"kubernetes.io/projected/db3c7d77-b4a0-4f87-9856-1e1a618db392-kube-api-access-mzs9n\") pod \"payload-processing-e2e-inf-a-80aa25-d8499f755-557zk\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:22.081081 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.081049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:22.354885 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.354860    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"]
Sep 10 20:52:22.355952 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:22.355923    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3c7d77_b4a0_4f87_9856_1e1a618db392.slice/crio-f550a2c582dd997ff9343af8c653031d07c5b5cf0ad7579807481d3a8000d104 WatchSource:0}: Error finding container f550a2c582dd997ff9343af8c653031d07c5b5cf0ad7579807481d3a8000d104: Status 404 returned error can't find the container with id f550a2c582dd997ff9343af8c653031d07c5b5cf0ad7579807481d3a8000d104
Sep 10 20:52:22.443752 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.443715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8029-50f346" event={"ID":"c39ef6ce-8e33-44c0-894a-e1e30ef2ff84","Type":"ContainerStarted","Data":"a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24"}
Sep 10 20:52:22.445437 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.445413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk" event={"ID":"db3c7d77-b4a0-4f87-9856-1e1a618db392","Type":"ContainerStarted","Data":"43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271"}
Sep 10 20:52:22.445576 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.445458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk" event={"ID":"db3c7d77-b4a0-4f87-9856-1e1a618db392","Type":"ContainerStarted","Data":"f550a2c582dd997ff9343af8c653031d07c5b5cf0ad7579807481d3a8000d104"}
Sep 10 20:52:22.445576 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.445515    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:22.469053 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.469001    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8029-50f346" podStartSLOduration=1.790854681 podStartE2EDuration="4.468980189s" totalImagesPullingTime="2.678125508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:22.464834662 +0000 UTC m=+978.375871947" watchObservedRunningTime="2026-09-10 20:52:22.468980189 +0000 UTC m=+978.380017479"
Sep 10 20:52:22.487464 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:22.487418    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk" podStartSLOduration=1.4874055959999999 podStartE2EDuration="1.487405596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:22.486832197 +0000 UTC m=+978.397869482" watchObservedRunningTime="2026-09-10 20:52:22.487405596 +0000 UTC m=+978.398442926"
Sep 10 20:52:23.057485 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.057383    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8029-50f346"]
Sep 10 20:52:23.060408 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.060377    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8029-50f346"]
Sep 10 20:52:23.175482 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.175452    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8029-952bbf"]
Sep 10 20:52:23.175935 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.175920    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" containerName="test-curl-8029-50f346"
Sep 10 20:52:23.175935 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.175936    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" containerName="test-curl-8029-50f346"
Sep 10 20:52:23.176027 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.175997    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" containerName="test-curl-8029-50f346"
Sep 10 20:52:23.179180 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.179165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:23.186205 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.186183    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8029-952bbf"]
Sep 10 20:52:23.320208 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.320143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4lm\" (UniqueName: \"kubernetes.io/projected/ff7509f8-4f3d-4c76-90b9-276fb863bb22-kube-api-access-lj4lm\") pod \"test-curl-8029-952bbf\" (UID: \"ff7509f8-4f3d-4c76-90b9-276fb863bb22\") " pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:23.421415 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.421385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4lm\" (UniqueName: \"kubernetes.io/projected/ff7509f8-4f3d-4c76-90b9-276fb863bb22-kube-api-access-lj4lm\") pod \"test-curl-8029-952bbf\" (UID: \"ff7509f8-4f3d-4c76-90b9-276fb863bb22\") " pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:23.429764 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.429744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4lm\" (UniqueName: \"kubernetes.io/projected/ff7509f8-4f3d-4c76-90b9-276fb863bb22-kube-api-access-lj4lm\") pod \"test-curl-8029-952bbf\" (UID: \"ff7509f8-4f3d-4c76-90b9-276fb863bb22\") " pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:23.450649 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:23.450630    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8029-50f346" container="test-curl-8029-50f346"
Sep 10 20:52:23.490111 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.490086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:23.764981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:23.764956    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8029-952bbf"]
Sep 10 20:52:23.766725 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:23.766700    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff7509f8_4f3d_4c76_90b9_276fb863bb22.slice/crio-3539c64650e6e170df31d9195939fdc6d89d5c8f9575f03dee5d86e69517d234 WatchSource:0}: Error finding container 3539c64650e6e170df31d9195939fdc6d89d5c8f9575f03dee5d86e69517d234: Status 404 returned error can't find the container with id 3539c64650e6e170df31d9195939fdc6d89d5c8f9575f03dee5d86e69517d234
Sep 10 20:52:24.457671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:24.457636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8029-952bbf" event={"ID":"ff7509f8-4f3d-4c76-90b9-276fb863bb22","Type":"ContainerStarted","Data":"385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa"}
Sep 10 20:52:24.458113 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:24.457707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8029-952bbf" event={"ID":"ff7509f8-4f3d-4c76-90b9-276fb863bb22","Type":"ContainerStarted","Data":"3539c64650e6e170df31d9195939fdc6d89d5c8f9575f03dee5d86e69517d234"}
Sep 10 20:52:24.458113 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:24.457729    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8029-50f346" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" containerName="test-curl-8029-50f346" containerID="cri-o://a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24" gracePeriod=2
Sep 10 20:52:24.479058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:24.479015    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8029-952bbf" podStartSLOduration=1.47900001 podStartE2EDuration="1.47900001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:24.477329163 +0000 UTC m=+980.388366448" watchObservedRunningTime="2026-09-10 20:52:24.47900001 +0000 UTC m=+980.390037295"
Sep 10 20:52:24.479828 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:24.479807    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-50f346\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:24.482356 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:24.482334    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-50f346\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:24.589165 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:24.589137    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-50f346\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:25.080492 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.080462    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8029-952bbf"]
Sep 10 20:52:25.084831 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.084797    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8029-952bbf"]
Sep 10 20:52:25.249119 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.249048    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"]
Sep 10 20:52:25.249736 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.249711    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" containerName="test-curl-8029-952bbf"
Sep 10 20:52:25.249736 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.249734    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" containerName="test-curl-8029-952bbf"
Sep 10 20:52:25.249879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.249843    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" containerName="test-curl-8029-952bbf"
Sep 10 20:52:25.253528 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.253506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.256484 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.256466    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c62a32a3-openshift-default-dockercfg-zbdg9\""
Sep 10 20:52:25.274481 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.274454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"]
Sep 10 20:52:25.439482 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-socket\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439644 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-podinfo\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439644 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istiod-ca-cert\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439644 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-certs\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439767 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-envoy\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439767 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-token\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439834 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-data\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439834 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-credential-socket\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.439941 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.439834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj2c\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-kube-api-access-tqj2c\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.463513 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:25.463488    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8029-952bbf" container="test-curl-8029-952bbf"
Sep 10 20:52:25.541454 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-socket\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541454 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-podinfo\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541646 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istiod-ca-cert\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541646 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-certs\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541646 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-envoy\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541825 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-token\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541825 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-data\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541825 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-credential-socket\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.541825 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj2c\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-kube-api-access-tqj2c\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.542039 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.541976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-certs\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.542100 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.542087    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-socket\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.542182 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.542148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-data\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.542346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.542319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istiod-ca-cert\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.542472 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.542320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-credential-socket\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.544176 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.544145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-envoy\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.544416 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.544396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-podinfo\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.551479 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.551456    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-token\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.552019 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.551999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj2c\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-kube-api-access-tqj2c\") pod \"e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") " pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.574279 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.574261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:25.856904 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.856876    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"]
Sep 10 20:52:25.858483 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:25.858447    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a62b045_3670_40d0_99df_5bdaf2ae58a6.slice/crio-d8c97f4fff9e749c57cd87373493de13388578bd29e43d4a76da83133b1ecc2b WatchSource:0}: Error finding container d8c97f4fff9e749c57cd87373493de13388578bd29e43d4a76da83133b1ecc2b: Status 404 returned error can't find the container with id d8c97f4fff9e749c57cd87373493de13388578bd29e43d4a76da83133b1ecc2b
Sep 10 20:52:25.860786 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.860745    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:52:25.860866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.860838    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:52:25.860907 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:25.860882    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:52:26.472326 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.472296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" event={"ID":"2a62b045-3670-40d0-99df-5bdaf2ae58a6","Type":"ContainerStarted","Data":"ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1"}
Sep 10 20:52:26.472671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.472351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" event={"ID":"2a62b045-3670-40d0-99df-5bdaf2ae58a6","Type":"ContainerStarted","Data":"d8c97f4fff9e749c57cd87373493de13388578bd29e43d4a76da83133b1ecc2b"}
Sep 10 20:52:26.472671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.472425    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8029-952bbf" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" containerName="test-curl-8029-952bbf" containerID="cri-o://385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa" gracePeriod=2
Sep 10 20:52:26.475244 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:26.475221    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-952bbf\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:26.498407 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.498369    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" podStartSLOduration=1.498355452 podStartE2EDuration="1.498355452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:26.497105074 +0000 UTC m=+982.408142361" watchObservedRunningTime="2026-09-10 20:52:26.498355452 +0000 UTC m=+982.409392736"
Sep 10 20:52:26.499676 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:26.499653    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-952bbf\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:26.575033 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.575002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:26.579875 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.579854    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:26.582774 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:26.582742    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-952bbf\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:26.639053 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.639029    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:26.753932 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.753864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd4xq\" (UniqueName: \"kubernetes.io/projected/c39ef6ce-8e33-44c0-894a-e1e30ef2ff84-kube-api-access-hd4xq\") pod \"c39ef6ce-8e33-44c0-894a-e1e30ef2ff84\" (UID: \"c39ef6ce-8e33-44c0-894a-e1e30ef2ff84\") "
Sep 10 20:52:26.756180 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.756148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39ef6ce-8e33-44c0-894a-e1e30ef2ff84-kube-api-access-hd4xq" (OuterVolumeSpecName: "kube-api-access-hd4xq") pod "c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" (UID: "c39ef6ce-8e33-44c0-894a-e1e30ef2ff84"). InnerVolumeSpecName "kube-api-access-hd4xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:52:26.854915 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:26.854891    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd4xq\" (UniqueName: \"kubernetes.io/projected/c39ef6ce-8e33-44c0-894a-e1e30ef2ff84-kube-api-access-hd4xq\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:52:27.479943 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:27.479907    2565 generic.go:356] "Generic (PLEG): container finished" podID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" containerID="a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24" exitCode=137
Sep 10 20:52:27.480368 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:27.479946    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:27.480368 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:27.480008    2565 scope.go:117] "RemoveContainer" containerID="a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24"
Sep 10 20:52:27.480368 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:27.480309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:27.481290 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:27.481271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:52:27.488974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:27.488960    2565 scope.go:117] "RemoveContainer" containerID="a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24"
Sep 10 20:52:27.489234 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:27.489217    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24\": container with ID starting with a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24 not found: ID does not exist" containerID="a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24"
Sep 10 20:52:27.489282 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:27.489244    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24"} err="failed to get container status \"a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24\": rpc error: code = NotFound desc = could not find container \"a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24\": container with ID starting with a6e4e49d656536f8a2902323b008e967b3ef0de4836537ff3cae46e8fb20bd24 not found: ID does not exist"
Sep 10 20:52:27.508917 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:27.508889    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-50f346\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" pod="opendatahub/test-curl-8029-50f346"
Sep 10 20:52:28.596233 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:28.596119    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c39ef6ce-8e33-44c0-894a-e1e30ef2ff84" path="/var/lib/kubelet/pods/c39ef6ce-8e33-44c0-894a-e1e30ef2ff84/volumes"
Sep 10 20:52:28.684972 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:28.684936    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:28.874125 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:28.874099    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj4lm\" (UniqueName: \"kubernetes.io/projected/ff7509f8-4f3d-4c76-90b9-276fb863bb22-kube-api-access-lj4lm\") pod \"ff7509f8-4f3d-4c76-90b9-276fb863bb22\" (UID: \"ff7509f8-4f3d-4c76-90b9-276fb863bb22\") "
Sep 10 20:52:28.876154 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:28.876131    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7509f8-4f3d-4c76-90b9-276fb863bb22-kube-api-access-lj4lm" (OuterVolumeSpecName: "kube-api-access-lj4lm") pod "ff7509f8-4f3d-4c76-90b9-276fb863bb22" (UID: "ff7509f8-4f3d-4c76-90b9-276fb863bb22"). InnerVolumeSpecName "kube-api-access-lj4lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:52:28.975180 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:28.975150    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4lm\" (UniqueName: \"kubernetes.io/projected/ff7509f8-4f3d-4c76-90b9-276fb863bb22-kube-api-access-lj4lm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:52:29.496815 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:29.496782    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" containerID="385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa" exitCode=137
Sep 10 20:52:29.496981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:29.496823    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:29.496981 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:29.496888    2565 scope.go:117] "RemoveContainer" containerID="385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa"
Sep 10 20:52:29.505914 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:29.505887    2565 scope.go:117] "RemoveContainer" containerID="385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa"
Sep 10 20:52:29.506216 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:29.506197    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa\": container with ID starting with 385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa not found: ID does not exist" containerID="385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa"
Sep 10 20:52:29.506283 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:29.506229    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa"} err="failed to get container status \"385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa\": rpc error: code = NotFound desc = could not find container \"385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa\": container with ID starting with 385b375f2734be0cc05769892fe0d008d42b47da824da59b754053aa83e429aa not found: ID does not exist"
Sep 10 20:52:29.508112 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:52:29.508089    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8029-952bbf\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" pod="opendatahub/test-curl-8029-952bbf"
Sep 10 20:52:30.591273 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:30.591241    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff7509f8-4f3d-4c76-90b9-276fb863bb22" path="/var/lib/kubelet/pods/ff7509f8-4f3d-4c76-90b9-276fb863bb22/volumes"
Sep 10 20:52:31.514265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.514237    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"]
Sep 10 20:52:31.518016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.517994    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.521297 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.521277    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5d9b44\""
Sep 10 20:52:31.530228 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.530067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"]
Sep 10 20:52:31.559246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.559221    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"]
Sep 10 20:52:31.562775 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.562759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:31.565957 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.565942    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5d9b44-dockercfg-8mwvx\""
Sep 10 20:52:31.573347 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.573328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"]
Sep 10 20:52:31.593150 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.593127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsj9w\" (UniqueName: \"kubernetes.io/projected/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-kube-api-access-wsj9w\") pod \"payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.593455 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.593183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.593455 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.593231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fm7\" (UniqueName: \"kubernetes.io/projected/f03947c7-ae70-4db8-8134-4360102262b5-kube-api-access-b9fm7\") pod \"payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") " pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:31.593455 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.593311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03947c7-ae70-4db8-8134-4360102262b5-plugins-config-volume\") pod \"payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") " pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:31.693805 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.693772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03947c7-ae70-4db8-8134-4360102262b5-plugins-config-volume\") pod \"payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") " pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:31.693980 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.693827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsj9w\" (UniqueName: \"kubernetes.io/projected/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-kube-api-access-wsj9w\") pod \"payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.693980 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.693858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.693980 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.693880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fm7\" (UniqueName: \"kubernetes.io/projected/f03947c7-ae70-4db8-8134-4360102262b5-kube-api-access-b9fm7\") pod \"payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") " pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:31.694534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.694513    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03947c7-ae70-4db8-8134-4360102262b5-plugins-config-volume\") pod \"payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") " pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:31.694576 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.694534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.702816 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.702794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fm7\" (UniqueName: \"kubernetes.io/projected/f03947c7-ae70-4db8-8134-4360102262b5-kube-api-access-b9fm7\") pod \"payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") " pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:31.703003 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.702983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsj9w\" (UniqueName: \"kubernetes.io/projected/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-kube-api-access-wsj9w\") pod \"payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.829348 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.829276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:31.873867 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:31.873839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:32.104775 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.104752    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"]
Sep 10 20:52:32.106230 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:32.106201    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd791bf_f11c_4a35_81cf_8cebd9dddb6d.slice/crio-b0393f21b37e3e516d7b2ad5682f3ef9aba7bc20237047ac407d61cabc9b752b WatchSource:0}: Error finding container b0393f21b37e3e516d7b2ad5682f3ef9aba7bc20237047ac407d61cabc9b752b: Status 404 returned error can't find the container with id b0393f21b37e3e516d7b2ad5682f3ef9aba7bc20237047ac407d61cabc9b752b
Sep 10 20:52:32.166030 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.166007    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"]
Sep 10 20:52:32.166295 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:32.166265    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03947c7_ae70_4db8_8134_4360102262b5.slice/crio-ca168856c71737df5b3ff8aa45f06dbfb216d6b22982cc5778c0b37be5eacd1e WatchSource:0}: Error finding container ca168856c71737df5b3ff8aa45f06dbfb216d6b22982cc5778c0b37be5eacd1e: Status 404 returned error can't find the container with id ca168856c71737df5b3ff8aa45f06dbfb216d6b22982cc5778c0b37be5eacd1e
Sep 10 20:52:32.522998 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.522896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs" event={"ID":"abd791bf-f11c-4a35-81cf-8cebd9dddb6d","Type":"ContainerStarted","Data":"efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5"}
Sep 10 20:52:32.522998 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.522959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs" event={"ID":"abd791bf-f11c-4a35-81cf-8cebd9dddb6d","Type":"ContainerStarted","Data":"b0393f21b37e3e516d7b2ad5682f3ef9aba7bc20237047ac407d61cabc9b752b"}
Sep 10 20:52:32.522998 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.522987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:32.524632 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.524604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g" event={"ID":"f03947c7-ae70-4db8-8134-4360102262b5","Type":"ContainerStarted","Data":"c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317"}
Sep 10 20:52:32.524750 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.524662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g" event={"ID":"f03947c7-ae70-4db8-8134-4360102262b5","Type":"ContainerStarted","Data":"ca168856c71737df5b3ff8aa45f06dbfb216d6b22982cc5778c0b37be5eacd1e"}
Sep 10 20:52:32.524750 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.524716    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:32.545591 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.545536    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs" podStartSLOduration=1.545517215 podStartE2EDuration="1.545517215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:32.54320363 +0000 UTC m=+988.454240915" watchObservedRunningTime="2026-09-10 20:52:32.545517215 +0000 UTC m=+988.456554503"
Sep 10 20:52:32.561777 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:32.561731    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g" podStartSLOduration=1.561716436 podStartE2EDuration="1.561716436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:32.559470026 +0000 UTC m=+988.470507322" watchObservedRunningTime="2026-09-10 20:52:32.561716436 +0000 UTC m=+988.472753719"
Sep 10 20:52:33.453185 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:33.453158    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:52:34.901200 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:34.901164    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:52:43.532865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:43.532833    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:52:43.533216 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:43.532909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:52:45.518084 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.518053    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"]
Sep 10 20:52:45.522093 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.522074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.525452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.525432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-467fc3-openshift-default-dockercfg-gwwgg\""
Sep 10 20:52:45.533832 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.533804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"]
Sep 10 20:52:45.621761 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.621723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-token\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.621761 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.621765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-data\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.621966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.621795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istiod-ca-cert\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.621966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.621828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-socket\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.621966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.621856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-certs\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.621966 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.621917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sj9h\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-kube-api-access-2sj9h\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.622145 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.622037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-envoy\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.622206 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.622178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-podinfo\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.622263 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.622235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-credential-socket\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.723775 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.723740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-data\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.723974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.723782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istiod-ca-cert\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.723974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.723832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-socket\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.723974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.723880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-certs\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.723974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.723920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sj9h\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-kube-api-access-2sj9h\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.723974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.723969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-envoy\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724210 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-podinfo\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724210 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-credential-socket\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724210 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-data\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724210 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-token\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724426 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-socket\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724426 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-certs\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724426 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-credential-socket\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.724648 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.724627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istiod-ca-cert\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.726593 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.726565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-envoy\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.726859 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.726839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-podinfo\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.734045 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.734014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-token\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.734348 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.734330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sj9h\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-kube-api-access-2sj9h\") pod \"e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") " pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:45.833050 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:45.832975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:46.110944 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.110919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"]
Sep 10 20:52:46.111850 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:46.111830    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2565bb7c_bc0d_4d2a_9bfc_756d671f9373.slice/crio-33c8af768da484e26550e3b8ff0651f75af3ae29916cfda1ef9998be0e13e706 WatchSource:0}: Error finding container 33c8af768da484e26550e3b8ff0651f75af3ae29916cfda1ef9998be0e13e706: Status 404 returned error can't find the container with id 33c8af768da484e26550e3b8ff0651f75af3ae29916cfda1ef9998be0e13e706
Sep 10 20:52:46.114189 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.114156    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:52:46.114286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.114233    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:52:46.114286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.114276    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:52:46.610555 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.610524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" event={"ID":"2565bb7c-bc0d-4d2a-9bfc-756d671f9373","Type":"ContainerStarted","Data":"3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a"}
Sep 10 20:52:46.610952 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.610568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" event={"ID":"2565bb7c-bc0d-4d2a-9bfc-756d671f9373","Type":"ContainerStarted","Data":"33c8af768da484e26550e3b8ff0651f75af3ae29916cfda1ef9998be0e13e706"}
Sep 10 20:52:46.633987 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.633929    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podStartSLOduration=1.6339095380000002 podStartE2EDuration="1.633909538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:46.633602196 +0000 UTC m=+1002.544639482" watchObservedRunningTime="2026-09-10 20:52:46.633909538 +0000 UTC m=+1002.544946831"
Sep 10 20:52:46.833641 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:46.833610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:47.839232 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:47.839177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:48.621754 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:48.621715    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:48.622496 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:48.622476    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:52:52.616167 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:52.616133    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:52:53.492261 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.492229    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"]
Sep 10 20:52:53.498845 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.498827    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"
Sep 10 20:52:53.502217 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.502059    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:52:53.502785 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.502573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:52:53.504296 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.503781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"]
Sep 10 20:52:53.600766 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.600739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8d73c-7295-44a1-90ff-5a2597659b9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx\" (UID: \"ffa8d73c-7295-44a1-90ff-5a2597659b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"
Sep 10 20:52:53.701259 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.701223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8d73c-7295-44a1-90ff-5a2597659b9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx\" (UID: \"ffa8d73c-7295-44a1-90ff-5a2597659b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"
Sep 10 20:52:53.702096 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.702070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8d73c-7295-44a1-90ff-5a2597659b9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx\" (UID: \"ffa8d73c-7295-44a1-90ff-5a2597659b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"
Sep 10 20:52:53.812435 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:53.812359    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"
Sep 10 20:52:54.108766 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:54.108737    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"]
Sep 10 20:52:54.111231 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:52:54.111207    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa8d73c_7295_44a1_90ff_5a2597659b9d.slice/crio-7d38de35d830a36279115194917305c4a72c705fb8f3f23712ca51c814b71690 WatchSource:0}: Error finding container 7d38de35d830a36279115194917305c4a72c705fb8f3f23712ca51c814b71690: Status 404 returned error can't find the container with id 7d38de35d830a36279115194917305c4a72c705fb8f3f23712ca51c814b71690
Sep 10 20:52:54.673437 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:54.673388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx" event={"ID":"ffa8d73c-7295-44a1-90ff-5a2597659b9d","Type":"ContainerStarted","Data":"7d38de35d830a36279115194917305c4a72c705fb8f3f23712ca51c814b71690"}
Sep 10 20:52:58.717494 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:58.717400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx" event={"ID":"ffa8d73c-7295-44a1-90ff-5a2597659b9d","Type":"ContainerStarted","Data":"e97ed30e4a81b829bf72603a120d3f54bacbb134b4f5a61b3c6bb35c39ecc337"}
Sep 10 20:52:58.736973 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:58.736917    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx" podStartSLOduration=1.764383269 podStartE2EDuration="5.73689694s" totalImagesPullingTime="3.972513671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:58.734333358 +0000 UTC m=+1014.645370671" watchObservedRunningTime="2026-09-10 20:52:58.73689694 +0000 UTC m=+1014.647934227"
Sep 10 20:52:59.724247 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:59.724208    2565 generic.go:356] "Generic (PLEG): container finished" podID="ffa8d73c-7295-44a1-90ff-5a2597659b9d" containerID="e97ed30e4a81b829bf72603a120d3f54bacbb134b4f5a61b3c6bb35c39ecc337" exitCode=0
Sep 10 20:52:59.724628 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:52:59.724293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx" event={"ID":"ffa8d73c-7295-44a1-90ff-5a2597659b9d","Type":"ContainerDied","Data":"e97ed30e4a81b829bf72603a120d3f54bacbb134b4f5a61b3c6bb35c39ecc337"}
Sep 10 20:53:00.890085 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:00.890049    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"
Sep 10 20:53:01.017515 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.017410    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8d73c-7295-44a1-90ff-5a2597659b9d-maas-api-service-ca\") pod \"ffa8d73c-7295-44a1-90ff-5a2597659b9d\" (UID: \"ffa8d73c-7295-44a1-90ff-5a2597659b9d\") "
Sep 10 20:53:01.017875 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.017849    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa8d73c-7295-44a1-90ff-5a2597659b9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffa8d73c-7295-44a1-90ff-5a2597659b9d" (UID: "ffa8d73c-7295-44a1-90ff-5a2597659b9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:53:01.119079 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.119046    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8d73c-7295-44a1-90ff-5a2597659b9d-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:01.739245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.739134    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerID="ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1" exitCode=0
Sep 10 20:53:01.739245 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.739214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" event={"ID":"2a62b045-3670-40d0-99df-5bdaf2ae58a6","Type":"ContainerDied","Data":"ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1"}
Sep 10 20:53:01.739736 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.739707    2565 scope.go:117] "RemoveContainer" containerID="ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1"
Sep 10 20:53:01.741120 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.741057    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:01.741239 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.741141    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:01.741239 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.741189    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:01.741721 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.741647    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx" event={"ID":"ffa8d73c-7295-44a1-90ff-5a2597659b9d","Type":"ContainerDied","Data":"7d38de35d830a36279115194917305c4a72c705fb8f3f23712ca51c814b71690"}
Sep 10 20:53:01.741721 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.741699    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"
Sep 10 20:53:01.741721 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:01.741710    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d38de35d830a36279115194917305c4a72c705fb8f3f23712ca51c814b71690"
Sep 10 20:53:02.762620 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:02.761602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" event={"ID":"2a62b045-3670-40d0-99df-5bdaf2ae58a6","Type":"ContainerStarted","Data":"72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18"}
Sep 10 20:53:03.575448 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.575413    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:53:03.575598 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.575506    2565 patch_prober.go:28] interesting pod/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:53:03.575598 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.575564    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 20:53:03.712013 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.711083    2565 patch_prober.go:28] interesting pod/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:33438->10.133.0.58:15021: read: connection reset by peer" start-of-body=
Sep 10 20:53:03.712013 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.711141    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:33438->10.133.0.58:15021: read: connection reset by peer"
Sep 10 20:53:03.771148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.771120    2565 generic.go:356] "Generic (PLEG): container finished" podID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerID="3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a" exitCode=0
Sep 10 20:53:03.771459 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.771189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" event={"ID":"2565bb7c-bc0d-4d2a-9bfc-756d671f9373","Type":"ContainerDied","Data":"3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a"}
Sep 10 20:53:03.771739 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.771721    2565 scope.go:117] "RemoveContainer" containerID="3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a"
Sep 10 20:53:03.772810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.772785    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:03.772929 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.772862    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:03.772929 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:03.772914    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:04.575099 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:04.575061    2565 patch_prober.go:28] interesting pod/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:53:04.575310 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:04.575125    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 20:53:04.787534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:04.787485    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" event={"ID":"2565bb7c-bc0d-4d2a-9bfc-756d671f9373","Type":"ContainerStarted","Data":"103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8"}
Sep 10 20:53:04.833340 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:04.833257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:53:04.833475 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:04.833401    2565 patch_prober.go:28] interesting pod/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 20:53:04.833475 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:04.833452    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 20:53:05.574839 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:05.574791    2565 patch_prober.go:28] interesting pod/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:53:05.575057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:05.574868    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 20:53:05.834386 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:05.834289    2565 patch_prober.go:28] interesting pod/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 20:53:05.834386 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:05.834364    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 20:53:06.582327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:06.581786    2565 patch_prober.go:28] interesting pod/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:53:06.582327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:06.581864    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 20:53:06.833826 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:06.833729    2565 patch_prober.go:28] interesting pod/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 20:53:06.833826 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:06.833805    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 20:53:07.575497 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:07.575452    2565 patch_prober.go:28] interesting pod/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:53:07.576039 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:07.575519    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 20:53:07.837912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:07.837803    2565 patch_prober.go:28] interesting pod/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 20:53:07.837912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:07.837875    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 20:53:08.579409 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:08.579371    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:53:08.579798 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:08.579653    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:53:08.580327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:08.580306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:53:08.834296 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:08.834202    2565 patch_prober.go:28] interesting pod/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 20:53:08.834296 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:08.834269    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 20:53:09.833644 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:09.833615    2565 patch_prober.go:28] interesting pod/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 20:53:09.834210 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:09.833663    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 20:53:10.292283 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.292134    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"]
Sep 10 20:53:10.294671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.294635    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c62a32a3-m9tgx"]
Sep 10 20:53:10.443125 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.443092    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"]
Sep 10 20:53:10.591332 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.591299    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa8d73c-7295-44a1-90ff-5a2597659b9d" path="/var/lib/kubelet/pods/ffa8d73c-7295-44a1-90ff-5a2597659b9d/volumes"
Sep 10 20:53:10.827004 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.826968    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy" containerID="cri-o://72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18" gracePeriod=30
Sep 10 20:53:10.838246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.838207    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:53:10.838640 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.838582    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:53:10.839569 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:10.839549    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:53:14.399491 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:14.399456    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:53:16.081912 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.081889    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:53:16.193501 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193430    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-token\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.193501 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-envoy\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.193680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193527    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-certs\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.193680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193551    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqj2c\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-kube-api-access-tqj2c\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.193680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193584    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-podinfo\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.193680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193604    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-socket\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.193680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193628    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-credential-socket\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.193680 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193659    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-data\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.194016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193706    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istiod-ca-cert\") pod \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\" (UID: \"2a62b045-3670-40d0-99df-5bdaf2ae58a6\") "
Sep 10 20:53:16.194016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193919    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:16.194016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.193995    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:16.194182 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.194145    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:16.194182 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.194170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:16.194297 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.194181    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.194297 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.194198    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-credential-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.194297 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.194231    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:53:16.195844 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.195816    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:53:16.195969 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.195913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 20:53:16.196020 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.195964    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-kube-api-access-tqj2c" (OuterVolumeSpecName: "kube-api-access-tqj2c") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "kube-api-access-tqj2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:53:16.196020 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.195994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a62b045-3670-40d0-99df-5bdaf2ae58a6" (UID: "2a62b045-3670-40d0-99df-5bdaf2ae58a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:16.294738 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.294711    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-envoy\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.294738 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.294731    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-workload-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.294738 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.294740    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqj2c\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-kube-api-access-tqj2c\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.294910 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.294749    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-podinfo\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.294910 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.294758    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-data\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.294910 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.294767    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istiod-ca-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.294910 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.294776    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b045-3670-40d0-99df-5bdaf2ae58a6-istio-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:16.879855 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.879753    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerID="72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18" exitCode=0
Sep 10 20:53:16.879855 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.879810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" event={"ID":"2a62b045-3670-40d0-99df-5bdaf2ae58a6","Type":"ContainerDied","Data":"72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18"}
Sep 10 20:53:16.880043 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.879923    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"
Sep 10 20:53:16.880265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.880239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk" event={"ID":"2a62b045-3670-40d0-99df-5bdaf2ae58a6","Type":"ContainerDied","Data":"d8c97f4fff9e749c57cd87373493de13388578bd29e43d4a76da83133b1ecc2b"}
Sep 10 20:53:16.880311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.880287    2565 scope.go:117] "RemoveContainer" containerID="72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18"
Sep 10 20:53:16.889383 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.889354    2565 scope.go:117] "RemoveContainer" containerID="ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1"
Sep 10 20:53:16.898932 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.898914    2565 scope.go:117] "RemoveContainer" containerID="72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18"
Sep 10 20:53:16.899210 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:53:16.899191    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18\": container with ID starting with 72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18 not found: ID does not exist" containerID="72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18"
Sep 10 20:53:16.899262 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.899218    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18"} err="failed to get container status \"72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18\": rpc error: code = NotFound desc = could not find container \"72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18\": container with ID starting with 72f3e5ff84fec609726f1e924c8d6d0b7f12cbeb65ec163560148ff5ca49df18 not found: ID does not exist"
Sep 10 20:53:16.899262 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.899236    2565 scope.go:117] "RemoveContainer" containerID="ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1"
Sep 10 20:53:16.899503 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:53:16.899487    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1\": container with ID starting with ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1 not found: ID does not exist" containerID="ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1"
Sep 10 20:53:16.899558 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.899506    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1"} err="failed to get container status \"ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1\": rpc error: code = NotFound desc = could not find container \"ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1\": container with ID starting with ef8fa24294e9f1d6e4a0c7464e0b4fe458c0774eb16f8f2ac83ef384530936c1 not found: ID does not exist"
Sep 10 20:53:16.904025 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.904003    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"]
Sep 10 20:53:16.908200 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:16.908179    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c62a32a3-openshift-default-7dff8c4c58-gw9tk"]
Sep 10 20:53:18.592373 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:18.592336    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" path="/var/lib/kubelet/pods/2a62b045-3670-40d0-99df-5bdaf2ae58a6/volumes"
Sep 10 20:53:31.603176 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:31.603142    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:53:38.700839 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.700808    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"]
Sep 10 20:53:38.701264 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701251    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy"
Sep 10 20:53:38.701308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701265    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy"
Sep 10 20:53:38.701308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701277    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa8d73c-7295-44a1-90ff-5a2597659b9d" containerName="revoke-keys"
Sep 10 20:53:38.701308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701282    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa8d73c-7295-44a1-90ff-5a2597659b9d" containerName="revoke-keys"
Sep 10 20:53:38.701308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701291    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy"
Sep 10 20:53:38.701308 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701297    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy"
Sep 10 20:53:38.701469 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701353    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa8d73c-7295-44a1-90ff-5a2597659b9d" containerName="revoke-keys"
Sep 10 20:53:38.701469 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701364    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy"
Sep 10 20:53:38.701469 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.701371    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a62b045-3670-40d0-99df-5bdaf2ae58a6" containerName="istio-proxy"
Sep 10 20:53:38.704528 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.704511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"
Sep 10 20:53:38.707679 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.707654    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:53:38.707800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.707711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:53:38.712574 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.712546    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"]
Sep 10 20:53:38.798006 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.797977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2bac-dc20-4e1d-922a-1013f510d976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs\" (UID: \"ed8f2bac-dc20-4e1d-922a-1013f510d976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"
Sep 10 20:53:38.899016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.898982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2bac-dc20-4e1d-922a-1013f510d976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs\" (UID: \"ed8f2bac-dc20-4e1d-922a-1013f510d976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"
Sep 10 20:53:38.899585 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:38.899566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2bac-dc20-4e1d-922a-1013f510d976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs\" (UID: \"ed8f2bac-dc20-4e1d-922a-1013f510d976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"
Sep 10 20:53:39.016493 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:39.016432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"
Sep 10 20:53:39.297136 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:39.297113    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"]
Sep 10 20:53:39.298522 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:53:39.298494    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8f2bac_dc20_4e1d_922a_1013f510d976.slice/crio-632933fc36973bea2fb3b436ac79f89361e560b95d0f90b24efffc8843fcc482 WatchSource:0}: Error finding container 632933fc36973bea2fb3b436ac79f89361e560b95d0f90b24efffc8843fcc482: Status 404 returned error can't find the container with id 632933fc36973bea2fb3b436ac79f89361e560b95d0f90b24efffc8843fcc482
Sep 10 20:53:40.010971 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:40.010936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs" event={"ID":"ed8f2bac-dc20-4e1d-922a-1013f510d976","Type":"ContainerStarted","Data":"f27a6d4976217fe7cea8ced6a1fb18d84c412a05c667ab7179fadeac1e417609"}
Sep 10 20:53:40.011407 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:40.010994    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs" event={"ID":"ed8f2bac-dc20-4e1d-922a-1013f510d976","Type":"ContainerStarted","Data":"632933fc36973bea2fb3b436ac79f89361e560b95d0f90b24efffc8843fcc482"}
Sep 10 20:53:40.028068 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:40.028022    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs" podStartSLOduration=2.028007426 podStartE2EDuration="2.028007426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:53:40.027319752 +0000 UTC m=+1055.938357036" watchObservedRunningTime="2026-09-10 20:53:40.028007426 +0000 UTC m=+1055.939044711"
Sep 10 20:53:41.018034 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:41.017999    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed8f2bac-dc20-4e1d-922a-1013f510d976" containerID="f27a6d4976217fe7cea8ced6a1fb18d84c412a05c667ab7179fadeac1e417609" exitCode=0
Sep 10 20:53:41.018395 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:41.018088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs" event={"ID":"ed8f2bac-dc20-4e1d-922a-1013f510d976","Type":"ContainerDied","Data":"f27a6d4976217fe7cea8ced6a1fb18d84c412a05c667ab7179fadeac1e417609"}
Sep 10 20:53:42.162247 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:42.162224    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"
Sep 10 20:53:42.331461 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:42.331379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2bac-dc20-4e1d-922a-1013f510d976-maas-api-service-ca\") pod \"ed8f2bac-dc20-4e1d-922a-1013f510d976\" (UID: \"ed8f2bac-dc20-4e1d-922a-1013f510d976\") "
Sep 10 20:53:42.331762 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:42.331739    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8f2bac-dc20-4e1d-922a-1013f510d976-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed8f2bac-dc20-4e1d-922a-1013f510d976" (UID: "ed8f2bac-dc20-4e1d-922a-1013f510d976"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:53:42.432938 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:42.432912    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2bac-dc20-4e1d-922a-1013f510d976-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:53:43.029416 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:43.029337    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"
Sep 10 20:53:43.029416 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:43.029373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs" event={"ID":"ed8f2bac-dc20-4e1d-922a-1013f510d976","Type":"ContainerDied","Data":"632933fc36973bea2fb3b436ac79f89361e560b95d0f90b24efffc8843fcc482"}
Sep 10 20:53:43.029585 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:43.029420    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632933fc36973bea2fb3b436ac79f89361e560b95d0f90b24efffc8843fcc482"
Sep 10 20:53:44.774965 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:44.774933    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"]
Sep 10 20:53:44.783250 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:44.783226    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-090c5b75-l95vs"]
Sep 10 20:53:45.665717 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.665664    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"]
Sep 10 20:53:45.666581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.666555    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8f2bac-dc20-4e1d-922a-1013f510d976" containerName="revoke-keys"
Sep 10 20:53:45.666581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.666581    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8f2bac-dc20-4e1d-922a-1013f510d976" containerName="revoke-keys"
Sep 10 20:53:45.666764 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.666737    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed8f2bac-dc20-4e1d-922a-1013f510d976" containerName="revoke-keys"
Sep 10 20:53:45.670130 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.670111    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.673355 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.673332    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-50a00efc-openshift-default-dockercfg-zdxtm\""
Sep 10 20:53:45.682477 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.682456    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"]
Sep 10 20:53:45.862313 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-envoy\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862313 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-data\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-certs\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-podinfo\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g522c\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-kube-api-access-g522c\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istiod-ca-cert\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-token\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-socket\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.862725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.862628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-credential-socket\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963279 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-credential-socket\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963279 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-envoy\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963279 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-data\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-certs\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-podinfo\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g522c\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-kube-api-access-g522c\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istiod-ca-cert\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-token\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-socket\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-credential-socket\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.963879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-data\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.964005 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.963915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-certs\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.964057 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.964035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-socket\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.964227 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.964210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istiod-ca-cert\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.965776 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.965752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-envoy\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.965918 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.965902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-podinfo\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.972168 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.972135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-token\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.972356 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.972340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g522c\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-kube-api-access-g522c\") pod \"e2e-ait-50a00efc-openshift-default-7985466476-dsl8h\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") " pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:45.980563 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:45.980545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:46.262562 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:46.262536    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"]
Sep 10 20:53:46.264357 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:53:46.264326    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cd302e_ce45_449d_93bf_78a6d71d45ef.slice/crio-6cfd6766406feb84e49286e8e9495b880d82174e0ccac9ef5db3cf3db4e03112 WatchSource:0}: Error finding container 6cfd6766406feb84e49286e8e9495b880d82174e0ccac9ef5db3cf3db4e03112: Status 404 returned error can't find the container with id 6cfd6766406feb84e49286e8e9495b880d82174e0ccac9ef5db3cf3db4e03112
Sep 10 20:53:46.266614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:46.266583    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:46.266703 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:46.266646    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:46.266703 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:46.266673    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:53:46.591515 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:46.591482    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed8f2bac-dc20-4e1d-922a-1013f510d976" path="/var/lib/kubelet/pods/ed8f2bac-dc20-4e1d-922a-1013f510d976/volumes"
Sep 10 20:53:47.058331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:47.058263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" event={"ID":"b3cd302e-ce45-449d-93bf-78a6d71d45ef","Type":"ContainerStarted","Data":"7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c"}
Sep 10 20:53:47.058331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:47.058312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" event={"ID":"b3cd302e-ce45-449d-93bf-78a6d71d45ef","Type":"ContainerStarted","Data":"6cfd6766406feb84e49286e8e9495b880d82174e0ccac9ef5db3cf3db4e03112"}
Sep 10 20:53:47.981209 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:47.981175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:47.986115 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:47.986088    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:48.009960 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:48.009912    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podStartSLOduration=3.009897016 podStartE2EDuration="3.009897016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:53:47.081206707 +0000 UTC m=+1062.992243992" watchObservedRunningTime="2026-09-10 20:53:48.009897016 +0000 UTC m=+1063.920934377"
Sep 10 20:53:48.064197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:48.064170    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:48.065048 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:48.065028    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:53:53.310011 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:53:53.309979    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:54:03.545897 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:03.545312    2565 patch_prober.go:28] interesting pod/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:44150->10.133.0.61:15021: read: connection reset by peer" start-of-body=
Sep 10 20:54:03.545897 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:03.545395    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:44150->10.133.0.61:15021: read: connection reset by peer"
Sep 10 20:54:04.167365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:04.167330    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerID="7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c" exitCode=0
Sep 10 20:54:04.167527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:04.167405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" event={"ID":"b3cd302e-ce45-449d-93bf-78a6d71d45ef","Type":"ContainerDied","Data":"7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c"}
Sep 10 20:54:04.167860 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:04.167846    2565 scope.go:117] "RemoveContainer" containerID="7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c"
Sep 10 20:54:04.168984 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:04.168958    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:54:04.169046 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:04.169019    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:54:04.169084 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:04.169048    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 20:54:05.176911 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:05.176879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" event={"ID":"b3cd302e-ce45-449d-93bf-78a6d71d45ef","Type":"ContainerStarted","Data":"818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9"}
Sep 10 20:54:05.981341 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:05.981304    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:54:05.981723 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:05.981666    2565 patch_prober.go:28] interesting pod/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 20:54:05.981935 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:05.981905    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 20:54:06.980962 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:06.980925    2565 patch_prober.go:28] interesting pod/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 20:54:06.981333 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:06.980982    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 20:54:07.980870 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:07.980839    2565 patch_prober.go:28] interesting pod/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 20:54:07.981044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:07.980892    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 20:54:08.981750 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:08.981715    2565 patch_prober.go:28] interesting pod/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 20:54:08.982118 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:08.981772    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 20:54:09.981490 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:09.981457    2565 patch_prober.go:28] interesting pod/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 20:54:09.981670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:09.981513    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 20:54:10.984733 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:10.984703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:54:10.985174 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:10.984996    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:54:10.985773 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:10.985754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:54:12.500171 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:12.500137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:54:13.591184 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.591151    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"]
Sep 10 20:54:13.596493 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.596465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.599798 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.599779    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5d9b44\"/\"kube-root-ca.crt\""
Sep 10 20:54:13.601469 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.601446    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5d9b44\"/\"default-dockercfg-qt9tn\""
Sep 10 20:54:13.601585 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.601466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5d9b44\"/\"openshift-service-ca.crt\""
Sep 10 20:54:13.601585 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.601493    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5d9b44\"/\"autores-model-5d9b44-kserve-self-signed-certs\""
Sep 10 20:54:13.606326 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.606302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"]
Sep 10 20:54:13.736532 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.736497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd4d\" (UniqueName: \"kubernetes.io/projected/f57f99e1-c638-4a0d-9721-05ff6706f0ea-kube-api-access-jtd4d\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.736747 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.736545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-model-cache\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.736747 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.736598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tls-certs\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.736747 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.736699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-home\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.736933 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.736768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-dshm\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.736933 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.736807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tmp-dir\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.837726 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.837667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tls-certs\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.838066 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.838040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-home\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.838235 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.838217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-dshm\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.838371 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.838355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tmp-dir\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.838494 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.838465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-home\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.839344 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.838487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd4d\" (UniqueName: \"kubernetes.io/projected/f57f99e1-c638-4a0d-9721-05ff6706f0ea-kube-api-access-jtd4d\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.839614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.839595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-model-cache\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.840157 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.838995    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tmp-dir\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.840157 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.840019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-model-cache\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.841500 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.841442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-dshm\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.841602 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.841532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tls-certs\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.847990 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.847963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd4d\" (UniqueName: \"kubernetes.io/projected/f57f99e1-c638-4a0d-9721-05ff6706f0ea-kube-api-access-jtd4d\") pod \"autores-model-5d9b44-kserve-8865f765f-n6q7n\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") " pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:13.908649 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:13.908616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:14.192086 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.192058    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"]
Sep 10 20:54:14.194153 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:54:14.194129    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf57f99e1_c638_4a0d_9721_05ff6706f0ea.slice/crio-a0d499b9f39b217e5248f43f5ca1e79248ab961966f40431dcdc23c5a584132e WatchSource:0}: Error finding container a0d499b9f39b217e5248f43f5ca1e79248ab961966f40431dcdc23c5a584132e: Status 404 returned error can't find the container with id a0d499b9f39b217e5248f43f5ca1e79248ab961966f40431dcdc23c5a584132e
Sep 10 20:54:14.230189 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.230164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n" event={"ID":"f57f99e1-c638-4a0d-9721-05ff6706f0ea","Type":"ContainerStarted","Data":"a0d499b9f39b217e5248f43f5ca1e79248ab961966f40431dcdc23c5a584132e"}
Sep 10 20:54:14.231363 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.231342    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"]
Sep 10 20:54:14.236166 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.236148    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"
Sep 10 20:54:14.239063 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.239035    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:54:14.239256 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.239224    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:54:14.244207 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.244184    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"]
Sep 10 20:54:14.346058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.346030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c433b16e-9b01-40a2-8d30-f4dac422c49d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50a00efc-m94jp\" (UID: \"c433b16e-9b01-40a2-8d30-f4dac422c49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"
Sep 10 20:54:14.447608 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.447510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c433b16e-9b01-40a2-8d30-f4dac422c49d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50a00efc-m94jp\" (UID: \"c433b16e-9b01-40a2-8d30-f4dac422c49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"
Sep 10 20:54:14.448212 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.448188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c433b16e-9b01-40a2-8d30-f4dac422c49d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50a00efc-m94jp\" (UID: \"c433b16e-9b01-40a2-8d30-f4dac422c49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"
Sep 10 20:54:14.564222 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.564194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"
Sep 10 20:54:14.847765 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:14.847712    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"]
Sep 10 20:54:15.239112 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:15.239079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp" event={"ID":"c433b16e-9b01-40a2-8d30-f4dac422c49d","Type":"ContainerStarted","Data":"9436b1b4e1fe9982faa68ba49625e347986d807bec254065572aa9ffe59edf7d"}
Sep 10 20:54:15.239255 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:15.239132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp" event={"ID":"c433b16e-9b01-40a2-8d30-f4dac422c49d","Type":"ContainerStarted","Data":"10fba0723e17f896d803dc818ccf172559b8e932ecc5776fa321ba1e4108a377"}
Sep 10 20:54:15.240928 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:15.240902    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n" event={"ID":"f57f99e1-c638-4a0d-9721-05ff6706f0ea","Type":"ContainerStarted","Data":"484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa"}
Sep 10 20:54:15.261116 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:15.261064    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp" podStartSLOduration=1.261047069 podStartE2EDuration="1.261047069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:54:15.258583632 +0000 UTC m=+1091.169620929" watchObservedRunningTime="2026-09-10 20:54:15.261047069 +0000 UTC m=+1091.172084353"
Sep 10 20:54:15.278814 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:15.278767    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n" podStartSLOduration=2.278753433 podStartE2EDuration="2.278753433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:54:15.27786471 +0000 UTC m=+1091.188901995" watchObservedRunningTime="2026-09-10 20:54:15.278753433 +0000 UTC m=+1091.189790717"
Sep 10 20:54:16.249482 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:16.249450    2565 generic.go:356] "Generic (PLEG): container finished" podID="c433b16e-9b01-40a2-8d30-f4dac422c49d" containerID="9436b1b4e1fe9982faa68ba49625e347986d807bec254065572aa9ffe59edf7d" exitCode=0
Sep 10 20:54:16.249864 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:16.249517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp" event={"ID":"c433b16e-9b01-40a2-8d30-f4dac422c49d","Type":"ContainerDied","Data":"9436b1b4e1fe9982faa68ba49625e347986d807bec254065572aa9ffe59edf7d"}
Sep 10 20:54:17.393799 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:17.393774    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"
Sep 10 20:54:17.473038 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:17.473008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c433b16e-9b01-40a2-8d30-f4dac422c49d-maas-api-service-ca\") pod \"c433b16e-9b01-40a2-8d30-f4dac422c49d\" (UID: \"c433b16e-9b01-40a2-8d30-f4dac422c49d\") "
Sep 10 20:54:17.473314 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:17.473291    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c433b16e-9b01-40a2-8d30-f4dac422c49d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c433b16e-9b01-40a2-8d30-f4dac422c49d" (UID: "c433b16e-9b01-40a2-8d30-f4dac422c49d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:54:17.574058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:17.573997    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c433b16e-9b01-40a2-8d30-f4dac422c49d-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:18.262318 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:18.262279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp" event={"ID":"c433b16e-9b01-40a2-8d30-f4dac422c49d","Type":"ContainerDied","Data":"10fba0723e17f896d803dc818ccf172559b8e932ecc5776fa321ba1e4108a377"}
Sep 10 20:54:18.262318 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:18.262311    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"
Sep 10 20:54:18.262525 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:18.262329    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10fba0723e17f896d803dc818ccf172559b8e932ecc5776fa321ba1e4108a377"
Sep 10 20:54:20.893309 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:20.893272    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"]
Sep 10 20:54:20.895229 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:20.895206    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50a00efc-m94jp"]
Sep 10 20:54:21.985106 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:21.985072    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"]
Sep 10 20:54:21.985560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:21.985274    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" containerID="cri-o://818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9" gracePeriod=30
Sep 10 20:54:22.592109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:22.592076    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c433b16e-9b01-40a2-8d30-f4dac422c49d" path="/var/lib/kubelet/pods/c433b16e-9b01-40a2-8d30-f4dac422c49d/volumes"
Sep 10 20:54:23.909344 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:23.909306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:23.909344 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:23.909352    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:23.921808 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:23.921783    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:24.316831 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:24.316750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:54:25.987400 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:25.987369    2565 patch_prober.go:28] interesting pod/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 20:54:25.987800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:25.987428    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 20:54:27.238731 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.238700    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:54:27.323560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.323482    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerID="818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9" exitCode=0
Sep 10 20:54:27.323560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.323529    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"
Sep 10 20:54:27.323772 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.323564    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" event={"ID":"b3cd302e-ce45-449d-93bf-78a6d71d45ef","Type":"ContainerDied","Data":"818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9"}
Sep 10 20:54:27.323772 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.323624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h" event={"ID":"b3cd302e-ce45-449d-93bf-78a6d71d45ef","Type":"ContainerDied","Data":"6cfd6766406feb84e49286e8e9495b880d82174e0ccac9ef5db3cf3db4e03112"}
Sep 10 20:54:27.323772 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.323657    2565 scope.go:117] "RemoveContainer" containerID="818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9"
Sep 10 20:54:27.333048 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.333031    2565 scope.go:117] "RemoveContainer" containerID="7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c"
Sep 10 20:54:27.341427 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.341409    2565 scope.go:117] "RemoveContainer" containerID="818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9"
Sep 10 20:54:27.341760 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:54:27.341736    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9\": container with ID starting with 818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9 not found: ID does not exist" containerID="818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9"
Sep 10 20:54:27.341835 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.341773    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9"} err="failed to get container status \"818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9\": rpc error: code = NotFound desc = could not find container \"818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9\": container with ID starting with 818dc4f7fb83d0157fc6e12265c252aefaf28ca4bc192e327fba54ac8ee148c9 not found: ID does not exist"
Sep 10 20:54:27.341835 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.341798    2565 scope.go:117] "RemoveContainer" containerID="7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c"
Sep 10 20:54:27.342090 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:54:27.342073    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c\": container with ID starting with 7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c not found: ID does not exist" containerID="7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c"
Sep 10 20:54:27.342142 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.342099    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c"} err="failed to get container status \"7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c\": rpc error: code = NotFound desc = could not find container \"7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c\": container with ID starting with 7755fa0cd3bce8e7d10bec7d9194acda70f50189ea4da9ce7e4940d5b5e0527c not found: ID does not exist"
Sep 10 20:54:27.367008 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.366987    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-credential-socket\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367026    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-podinfo\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istiod-ca-cert\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367073    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-certs\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-data\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-socket\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367205    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-envoy\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367374 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367241    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g522c\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-kube-api-access-g522c\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367374 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367265    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:54:27.367374 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367319    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-token\") pod \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\" (UID: \"b3cd302e-ce45-449d-93bf-78a6d71d45ef\") "
Sep 10 20:54:27.367518 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367404    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:54:27.367577 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367506    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:54:27.367577 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367546    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:54:27.367768 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367750    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-credential-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.367840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367779    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istiod-ca-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.367840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367795    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.367840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367809    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-workload-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.367840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.367747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:54:27.369493 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.369466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-kube-api-access-g522c" (OuterVolumeSpecName: "kube-api-access-g522c") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "kube-api-access-g522c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:54:27.369493 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.369479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:54:27.369602 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.369511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 20:54:27.369670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.369653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3cd302e-ce45-449d-93bf-78a6d71d45ef" (UID: "b3cd302e-ce45-449d-93bf-78a6d71d45ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:54:27.469151 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.469126    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-podinfo\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.469151 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.469148    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-data\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.469270 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.469158    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-envoy\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.469270 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.469166    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g522c\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-kube-api-access-g522c\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.469270 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.469174    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd302e-ce45-449d-93bf-78a6d71d45ef-istio-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:27.648192 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.648165    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"]
Sep 10 20:54:27.652219 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:27.652199    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-50a00efc-openshift-default-7985466476-dsl8h"]
Sep 10 20:54:28.591675 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:28.591642    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" path="/var/lib/kubelet/pods/b3cd302e-ce45-449d-93bf-78a6d71d45ef/volumes"
Sep 10 20:54:32.707519 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:32.707477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:54:47.140741 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.140705    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"]
Sep 10 20:54:47.141361 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141339    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy"
Sep 10 20:54:47.141361 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141362    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy"
Sep 10 20:54:47.141503 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141373    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c433b16e-9b01-40a2-8d30-f4dac422c49d" containerName="revoke-keys"
Sep 10 20:54:47.141503 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141381    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c433b16e-9b01-40a2-8d30-f4dac422c49d" containerName="revoke-keys"
Sep 10 20:54:47.141503 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141402    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy"
Sep 10 20:54:47.141503 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141410    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy"
Sep 10 20:54:47.141737 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141555    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy"
Sep 10 20:54:47.141737 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141567    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c433b16e-9b01-40a2-8d30-f4dac422c49d" containerName="revoke-keys"
Sep 10 20:54:47.141737 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.141580    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3cd302e-ce45-449d-93bf-78a6d71d45ef" containerName="istio-proxy"
Sep 10 20:54:47.145173 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.145152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"
Sep 10 20:54:47.149022 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.148994    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:54:47.149197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.149169    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:54:47.150418 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.150398    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"]
Sep 10 20:54:47.248986 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.248949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/868eb515-7e84-4dc2-91e3-e23ad7a5846a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl\" (UID: \"868eb515-7e84-4dc2-91e3-e23ad7a5846a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"
Sep 10 20:54:47.350240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.350212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/868eb515-7e84-4dc2-91e3-e23ad7a5846a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl\" (UID: \"868eb515-7e84-4dc2-91e3-e23ad7a5846a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"
Sep 10 20:54:47.350930 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.350906    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/868eb515-7e84-4dc2-91e3-e23ad7a5846a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl\" (UID: \"868eb515-7e84-4dc2-91e3-e23ad7a5846a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"
Sep 10 20:54:47.457455 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.457365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"
Sep 10 20:54:47.728638 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:47.728612    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"]
Sep 10 20:54:47.729596 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:54:47.729573    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868eb515_7e84_4dc2_91e3_e23ad7a5846a.slice/crio-9463ed24dc784ad4ee69890c8c180a5d96060a4881013253d664814a126ef0be WatchSource:0}: Error finding container 9463ed24dc784ad4ee69890c8c180a5d96060a4881013253d664814a126ef0be: Status 404 returned error can't find the container with id 9463ed24dc784ad4ee69890c8c180a5d96060a4881013253d664814a126ef0be
Sep 10 20:54:48.454052 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:48.454018    2565 generic.go:356] "Generic (PLEG): container finished" podID="868eb515-7e84-4dc2-91e3-e23ad7a5846a" containerID="2106647d2efff475b7cd99bdc69badd9008b3af0a6e14559c6ce93f5b964b534" exitCode=0
Sep 10 20:54:48.454410 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:48.454100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl" event={"ID":"868eb515-7e84-4dc2-91e3-e23ad7a5846a","Type":"ContainerDied","Data":"2106647d2efff475b7cd99bdc69badd9008b3af0a6e14559c6ce93f5b964b534"}
Sep 10 20:54:48.454410 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:48.454206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl" event={"ID":"868eb515-7e84-4dc2-91e3-e23ad7a5846a","Type":"ContainerStarted","Data":"9463ed24dc784ad4ee69890c8c180a5d96060a4881013253d664814a126ef0be"}
Sep 10 20:54:49.616972 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:49.616935    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:54:49.621895 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:49.621875    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"
Sep 10 20:54:49.672010 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:49.671985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/868eb515-7e84-4dc2-91e3-e23ad7a5846a-maas-api-service-ca\") pod \"868eb515-7e84-4dc2-91e3-e23ad7a5846a\" (UID: \"868eb515-7e84-4dc2-91e3-e23ad7a5846a\") "
Sep 10 20:54:49.672304 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:49.672280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868eb515-7e84-4dc2-91e3-e23ad7a5846a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "868eb515-7e84-4dc2-91e3-e23ad7a5846a" (UID: "868eb515-7e84-4dc2-91e3-e23ad7a5846a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:54:49.773635 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:49.773579    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/868eb515-7e84-4dc2-91e3-e23ad7a5846a-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:50.467257 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:50.467226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl" event={"ID":"868eb515-7e84-4dc2-91e3-e23ad7a5846a","Type":"ContainerDied","Data":"9463ed24dc784ad4ee69890c8c180a5d96060a4881013253d664814a126ef0be"}
Sep 10 20:54:50.467257 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:50.467246    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"
Sep 10 20:54:50.467473 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:50.467273    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9463ed24dc784ad4ee69890c8c180a5d96060a4881013253d664814a126ef0be"
Sep 10 20:54:53.310847 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.310818    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"]
Sep 10 20:54:53.311274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.311261    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="868eb515-7e84-4dc2-91e3-e23ad7a5846a" containerName="revoke-keys"
Sep 10 20:54:53.311327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.311276    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="868eb515-7e84-4dc2-91e3-e23ad7a5846a" containerName="revoke-keys"
Sep 10 20:54:53.311363 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.311337    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="868eb515-7e84-4dc2-91e3-e23ad7a5846a" containerName="revoke-keys"
Sep 10 20:54:53.314564 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.314547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"
Sep 10 20:54:53.320117 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.320093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:54:53.320551 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.320359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:54:53.321304 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.321281    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"]
Sep 10 20:54:53.402904 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.402874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51998dc2-37b6-4f6d-aabd-936495516224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt\" (UID: \"51998dc2-37b6-4f6d-aabd-936495516224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"
Sep 10 20:54:53.504020 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.503990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51998dc2-37b6-4f6d-aabd-936495516224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt\" (UID: \"51998dc2-37b6-4f6d-aabd-936495516224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"
Sep 10 20:54:53.504541 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.504523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51998dc2-37b6-4f6d-aabd-936495516224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt\" (UID: \"51998dc2-37b6-4f6d-aabd-936495516224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"
Sep 10 20:54:53.627357 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.627330    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"
Sep 10 20:54:53.898752 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:53.898728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"]
Sep 10 20:54:53.900149 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:54:53.900123    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51998dc2_37b6_4f6d_aabd_936495516224.slice/crio-22a08bbc5e9fb7a2edca3c77026a86750d9741eae0fa9d821029b63b24644ca8 WatchSource:0}: Error finding container 22a08bbc5e9fb7a2edca3c77026a86750d9741eae0fa9d821029b63b24644ca8: Status 404 returned error can't find the container with id 22a08bbc5e9fb7a2edca3c77026a86750d9741eae0fa9d821029b63b24644ca8
Sep 10 20:54:54.490452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:54.490360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt" event={"ID":"51998dc2-37b6-4f6d-aabd-936495516224","Type":"ContainerStarted","Data":"a1251715bca59fba69f1842569a732916c4ab96874afbd285dbd074ee872a8b0"}
Sep 10 20:54:54.490452 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:54.490424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt" event={"ID":"51998dc2-37b6-4f6d-aabd-936495516224","Type":"ContainerStarted","Data":"22a08bbc5e9fb7a2edca3c77026a86750d9741eae0fa9d821029b63b24644ca8"}
Sep 10 20:54:54.508402 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:54.508361    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt" podStartSLOduration=1.508347143 podStartE2EDuration="1.508347143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:54:54.506864339 +0000 UTC m=+1130.417901624" watchObservedRunningTime="2026-09-10 20:54:54.508347143 +0000 UTC m=+1130.419384500"
Sep 10 20:54:55.498155 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:55.498119    2565 generic.go:356] "Generic (PLEG): container finished" podID="51998dc2-37b6-4f6d-aabd-936495516224" containerID="a1251715bca59fba69f1842569a732916c4ab96874afbd285dbd074ee872a8b0" exitCode=0
Sep 10 20:54:55.498520 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:55.498199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt" event={"ID":"51998dc2-37b6-4f6d-aabd-936495516224","Type":"ContainerDied","Data":"a1251715bca59fba69f1842569a732916c4ab96874afbd285dbd074ee872a8b0"}
Sep 10 20:54:56.635375 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:56.635341    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"
Sep 10 20:54:56.730765 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:56.730734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51998dc2-37b6-4f6d-aabd-936495516224-maas-api-service-ca\") pod \"51998dc2-37b6-4f6d-aabd-936495516224\" (UID: \"51998dc2-37b6-4f6d-aabd-936495516224\") "
Sep 10 20:54:56.731072 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:56.731049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51998dc2-37b6-4f6d-aabd-936495516224-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51998dc2-37b6-4f6d-aabd-936495516224" (UID: "51998dc2-37b6-4f6d-aabd-936495516224"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:54:56.832488 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:56.832406    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51998dc2-37b6-4f6d-aabd-936495516224-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:57.510957 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:57.510927    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"
Sep 10 20:54:57.511133 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:57.510969    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt" event={"ID":"51998dc2-37b6-4f6d-aabd-936495516224","Type":"ContainerDied","Data":"22a08bbc5e9fb7a2edca3c77026a86750d9741eae0fa9d821029b63b24644ca8"}
Sep 10 20:54:57.511133 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:57.511013    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22a08bbc5e9fb7a2edca3c77026a86750d9741eae0fa9d821029b63b24644ca8"
Sep 10 20:54:58.178626 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.178593    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"]
Sep 10 20:54:58.179029 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.178852    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g" podUID="f03947c7-ae70-4db8-8134-4360102262b5" containerName="payload-processing" containerID="cri-o://c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317" gracePeriod=30
Sep 10 20:54:58.185437 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.185361    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"]
Sep 10 20:54:58.185563 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.185541    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs" podUID="abd791bf-f11c-4a35-81cf-8cebd9dddb6d" containerName="payload-pre-processing" containerID="cri-o://efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5" gracePeriod=30
Sep 10 20:54:58.410012 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.409969    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"]
Sep 10 20:54:58.414557 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.414532    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5d9b44-4hdxl"]
Sep 10 20:54:58.475763 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.475735    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:54:58.478743 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.478724    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:54:58.519238 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.519205    2565 generic.go:356] "Generic (PLEG): container finished" podID="abd791bf-f11c-4a35-81cf-8cebd9dddb6d" containerID="efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5" exitCode=0
Sep 10 20:54:58.519404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.519244    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"
Sep 10 20:54:58.519404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.519279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs" event={"ID":"abd791bf-f11c-4a35-81cf-8cebd9dddb6d","Type":"ContainerDied","Data":"efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5"}
Sep 10 20:54:58.519404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.519360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs" event={"ID":"abd791bf-f11c-4a35-81cf-8cebd9dddb6d","Type":"ContainerDied","Data":"b0393f21b37e3e516d7b2ad5682f3ef9aba7bc20237047ac407d61cabc9b752b"}
Sep 10 20:54:58.519404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.519388    2565 scope.go:117] "RemoveContainer" containerID="efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5"
Sep 10 20:54:58.520876 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.520853    2565 generic.go:356] "Generic (PLEG): container finished" podID="f03947c7-ae70-4db8-8134-4360102262b5" containerID="c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317" exitCode=0
Sep 10 20:54:58.521004 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.520905    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"
Sep 10 20:54:58.521004 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.520936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g" event={"ID":"f03947c7-ae70-4db8-8134-4360102262b5","Type":"ContainerDied","Data":"c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317"}
Sep 10 20:54:58.521004 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.520977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g" event={"ID":"f03947c7-ae70-4db8-8134-4360102262b5","Type":"ContainerDied","Data":"ca168856c71737df5b3ff8aa45f06dbfb216d6b22982cc5778c0b37be5eacd1e"}
Sep 10 20:54:58.530362 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.530346    2565 scope.go:117] "RemoveContainer" containerID="efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5"
Sep 10 20:54:58.530666 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:54:58.530644    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5\": container with ID starting with efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5 not found: ID does not exist" containerID="efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5"
Sep 10 20:54:58.530744 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.530699    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5"} err="failed to get container status \"efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5\": rpc error: code = NotFound desc = could not find container \"efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5\": container with ID starting with efc5779b425d2bddaa4c8b7e5cfb9d6030e9b9a02bb1f799630e285573106af5 not found: ID does not exist"
Sep 10 20:54:58.530744 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.530722    2565 scope.go:117] "RemoveContainer" containerID="c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317"
Sep 10 20:54:58.540974 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.540942    2565 scope.go:117] "RemoveContainer" containerID="c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317"
Sep 10 20:54:58.541289 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:54:58.541268    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317\": container with ID starting with c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317 not found: ID does not exist" containerID="c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317"
Sep 10 20:54:58.541404 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.541294    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317"} err="failed to get container status \"c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317\": rpc error: code = NotFound desc = could not find container \"c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317\": container with ID starting with c1007d8550e21c5beb3d807f909e04bb2e464a8976d62462ceda544527397317 not found: ID does not exist"
Sep 10 20:54:58.549580 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.549559    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-plugins-config-volume\") pod \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") "
Sep 10 20:54:58.549718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.549597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9fm7\" (UniqueName: \"kubernetes.io/projected/f03947c7-ae70-4db8-8134-4360102262b5-kube-api-access-b9fm7\") pod \"f03947c7-ae70-4db8-8134-4360102262b5\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") "
Sep 10 20:54:58.549788 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.549743    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsj9w\" (UniqueName: \"kubernetes.io/projected/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-kube-api-access-wsj9w\") pod \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\" (UID: \"abd791bf-f11c-4a35-81cf-8cebd9dddb6d\") "
Sep 10 20:54:58.549866 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.549808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03947c7-ae70-4db8-8134-4360102262b5-plugins-config-volume\") pod \"f03947c7-ae70-4db8-8134-4360102262b5\" (UID: \"f03947c7-ae70-4db8-8134-4360102262b5\") "
Sep 10 20:54:58.550040 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.550013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abd791bf-f11c-4a35-81cf-8cebd9dddb6d" (UID: "abd791bf-f11c-4a35-81cf-8cebd9dddb6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:54:58.550227 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.550212    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-plugins-config-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:58.550346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.550321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03947c7-ae70-4db8-8134-4360102262b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f03947c7-ae70-4db8-8134-4360102262b5" (UID: "f03947c7-ae70-4db8-8134-4360102262b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:54:58.552028 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.551938    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03947c7-ae70-4db8-8134-4360102262b5-kube-api-access-b9fm7" (OuterVolumeSpecName: "kube-api-access-b9fm7") pod "f03947c7-ae70-4db8-8134-4360102262b5" (UID: "f03947c7-ae70-4db8-8134-4360102262b5"). InnerVolumeSpecName "kube-api-access-b9fm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:54:58.552323 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.552290    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-kube-api-access-wsj9w" (OuterVolumeSpecName: "kube-api-access-wsj9w") pod "abd791bf-f11c-4a35-81cf-8cebd9dddb6d" (UID: "abd791bf-f11c-4a35-81cf-8cebd9dddb6d"). InnerVolumeSpecName "kube-api-access-wsj9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:54:58.591717 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.591680    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="868eb515-7e84-4dc2-91e3-e23ad7a5846a" path="/var/lib/kubelet/pods/868eb515-7e84-4dc2-91e3-e23ad7a5846a/volumes"
Sep 10 20:54:58.650741 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.650718    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03947c7-ae70-4db8-8134-4360102262b5-plugins-config-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:58.650831 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.650742    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9fm7\" (UniqueName: \"kubernetes.io/projected/f03947c7-ae70-4db8-8134-4360102262b5-kube-api-access-b9fm7\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:58.650831 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.650757    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsj9w\" (UniqueName: \"kubernetes.io/projected/abd791bf-f11c-4a35-81cf-8cebd9dddb6d-kube-api-access-wsj9w\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:54:58.838120 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.838099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"]
Sep 10 20:54:58.844274 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.844244    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5d9b44-59f7574b4b-nncfs"]
Sep 10 20:54:58.857806 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.857779    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"]
Sep 10 20:54:58.861592 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:54:58.861572    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5d9b44-6b599f49c-6m92g"]
Sep 10 20:55:00.591339 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:00.591306    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd791bf-f11c-4a35-81cf-8cebd9dddb6d" path="/var/lib/kubelet/pods/abd791bf-f11c-4a35-81cf-8cebd9dddb6d/volumes"
Sep 10 20:55:00.591731 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:00.591715    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03947c7-ae70-4db8-8134-4360102262b5" path="/var/lib/kubelet/pods/f03947c7-ae70-4db8-8134-4360102262b5/volumes"
Sep 10 20:55:00.921423 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:00.921342    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"]
Sep 10 20:55:00.924943 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:00.924911    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1672bb6-7zzmt"]
Sep 10 20:55:02.590818 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:02.590780    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51998dc2-37b6-4f6d-aabd-936495516224" path="/var/lib/kubelet/pods/51998dc2-37b6-4f6d-aabd-936495516224/volumes"
Sep 10 20:55:03.754720 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:03.754677    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"]
Sep 10 20:55:06.899492 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:06.899458    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:55:18.773681 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:18.773642    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n" podUID="f57f99e1-c638-4a0d-9721-05ff6706f0ea" containerName="main" containerID="cri-o://484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa" gracePeriod=45
Sep 10 20:55:19.022385 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.022360    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:55:19.144175 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-model-cache\") pod \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") "
Sep 10 20:55:19.144346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-home\") pod \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") "
Sep 10 20:55:19.144346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144246    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tls-certs\") pod \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") "
Sep 10 20:55:19.144346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144278    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtd4d\" (UniqueName: \"kubernetes.io/projected/f57f99e1-c638-4a0d-9721-05ff6706f0ea-kube-api-access-jtd4d\") pod \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") "
Sep 10 20:55:19.144346 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144303    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-dshm\") pod \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") "
Sep 10 20:55:19.144579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144361    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tmp-dir\") pod \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\" (UID: \"f57f99e1-c638-4a0d-9721-05ff6706f0ea\") "
Sep 10 20:55:19.144579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144428    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "f57f99e1-c638-4a0d-9721-05ff6706f0ea" (UID: "f57f99e1-c638-4a0d-9721-05ff6706f0ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.144579 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144518    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-home" (OuterVolumeSpecName: "home") pod "f57f99e1-c638-4a0d-9721-05ff6706f0ea" (UID: "f57f99e1-c638-4a0d-9721-05ff6706f0ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.144754 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f57f99e1-c638-4a0d-9721-05ff6706f0ea" (UID: "f57f99e1-c638-4a0d-9721-05ff6706f0ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.144813 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144790    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.144813 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.144809    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.146508 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.146482    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f57f99e1-c638-4a0d-9721-05ff6706f0ea" (UID: "f57f99e1-c638-4a0d-9721-05ff6706f0ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:55:19.146621 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.146540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57f99e1-c638-4a0d-9721-05ff6706f0ea-kube-api-access-jtd4d" (OuterVolumeSpecName: "kube-api-access-jtd4d") pod "f57f99e1-c638-4a0d-9721-05ff6706f0ea" (UID: "f57f99e1-c638-4a0d-9721-05ff6706f0ea"). InnerVolumeSpecName "kube-api-access-jtd4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:55:19.146751 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.146725    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-dshm" (OuterVolumeSpecName: "dshm") pod "f57f99e1-c638-4a0d-9721-05ff6706f0ea" (UID: "f57f99e1-c638-4a0d-9721-05ff6706f0ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.245377 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.245353    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.245489 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.245381    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.245489 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.245397    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57f99e1-c638-4a0d-9721-05ff6706f0ea-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.245489 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.245410    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd4d\" (UniqueName: \"kubernetes.io/projected/f57f99e1-c638-4a0d-9721-05ff6706f0ea-kube-api-access-jtd4d\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.645210 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.645179    2565 generic.go:356] "Generic (PLEG): container finished" podID="f57f99e1-c638-4a0d-9721-05ff6706f0ea" containerID="484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa" exitCode=0
Sep 10 20:55:19.645365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.645214    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"
Sep 10 20:55:19.645365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.645238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n" event={"ID":"f57f99e1-c638-4a0d-9721-05ff6706f0ea","Type":"ContainerDied","Data":"484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa"}
Sep 10 20:55:19.645365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.645268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n" event={"ID":"f57f99e1-c638-4a0d-9721-05ff6706f0ea","Type":"ContainerDied","Data":"a0d499b9f39b217e5248f43f5ca1e79248ab961966f40431dcdc23c5a584132e"}
Sep 10 20:55:19.645365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.645283    2565 scope.go:117] "RemoveContainer" containerID="484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa"
Sep 10 20:55:19.654761 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.654745    2565 scope.go:117] "RemoveContainer" containerID="484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa"
Sep 10 20:55:19.655029 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:55:19.655010    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa\": container with ID starting with 484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa not found: ID does not exist" containerID="484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa"
Sep 10 20:55:19.655109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.655039    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa"} err="failed to get container status \"484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa\": rpc error: code = NotFound desc = could not find container \"484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa\": container with ID starting with 484a343fbd40a8467eaa8d4d454d6011456e2944d8ee0a5ef44824a5988e18fa not found: ID does not exist"
Sep 10 20:55:19.669954 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.669932    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"]
Sep 10 20:55:19.673810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:19.673780    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5d9b44/autores-model-5d9b44-kserve-8865f765f-n6q7n"]
Sep 10 20:55:20.294967 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:20.294931    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:55:20.590842 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:20.590804    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f57f99e1-c638-4a0d-9721-05ff6706f0ea" path="/var/lib/kubelet/pods/f57f99e1-c638-4a0d-9721-05ff6706f0ea/volumes"
Sep 10 20:55:37.099140 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:37.099063    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:55:47.690286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.690250    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"]
Sep 10 20:55:47.691309 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691275    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f57f99e1-c638-4a0d-9721-05ff6706f0ea" containerName="main"
Sep 10 20:55:47.691309 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691307    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f57f99e1-c638-4a0d-9721-05ff6706f0ea" containerName="main"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691321    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03947c7-ae70-4db8-8134-4360102262b5" containerName="payload-processing"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691328    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03947c7-ae70-4db8-8134-4360102262b5" containerName="payload-processing"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691342    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd791bf-f11c-4a35-81cf-8cebd9dddb6d" containerName="payload-pre-processing"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691349    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd791bf-f11c-4a35-81cf-8cebd9dddb6d" containerName="payload-pre-processing"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691363    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51998dc2-37b6-4f6d-aabd-936495516224" containerName="revoke-keys"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691368    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51998dc2-37b6-4f6d-aabd-936495516224" containerName="revoke-keys"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691445    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd791bf-f11c-4a35-81cf-8cebd9dddb6d" containerName="payload-pre-processing"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691453    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f57f99e1-c638-4a0d-9721-05ff6706f0ea" containerName="main"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691459    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03947c7-ae70-4db8-8134-4360102262b5" containerName="payload-processing"
Sep 10 20:55:47.691505 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.691467    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51998dc2-37b6-4f6d-aabd-936495516224" containerName="revoke-keys"
Sep 10 20:55:47.694742 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.694720    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.697679 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.697653    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-75dcbb-serving-cert\""
Sep 10 20:55:47.697785 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.697656    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 20:55:47.697785 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.697723    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:55:47.699085 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.699064    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqzh2\""
Sep 10 20:55:47.699247 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.699227    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 20:55:47.706098 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.706077    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"]
Sep 10 20:55:47.815432 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.815393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qh4m\" (UniqueName: \"kubernetes.io/projected/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-kube-api-access-6qh4m\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.815591 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.815434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-metrics-certs\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.815591 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.815532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-maas-api-tls\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.815591 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.815564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.916808 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.916770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-maas-api-tls\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.916989 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.916813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.916989 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.916909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qh4m\" (UniqueName: \"kubernetes.io/projected/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-kube-api-access-6qh4m\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.916989 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.916936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-metrics-certs\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.917537 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.917508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.919306 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.919284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-maas-api-tls\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.919437 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.919306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-metrics-certs\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:47.925667 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:47.925644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qh4m\" (UniqueName: \"kubernetes.io/projected/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-kube-api-access-6qh4m\") pod \"maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:48.006718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:48.006641    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:48.283817 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:48.283793    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"]
Sep 10 20:55:48.285761 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:55:48.285733    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cc4280_8228_453d_aa4b_98c8b86fa4cd.slice/crio-0a9a2b8177e0e5390c0e8879c383324b85d22aac4f73ee0c4dcbb087a445317b WatchSource:0}: Error finding container 0a9a2b8177e0e5390c0e8879c383324b85d22aac4f73ee0c4dcbb087a445317b: Status 404 returned error can't find the container with id 0a9a2b8177e0e5390c0e8879c383324b85d22aac4f73ee0c4dcbb087a445317b
Sep 10 20:55:48.287044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:48.287029    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:55:48.813010 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:48.812972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq" event={"ID":"f2cc4280-8228-453d-aa4b-98c8b86fa4cd","Type":"ContainerStarted","Data":"0a9a2b8177e0e5390c0e8879c383324b85d22aac4f73ee0c4dcbb087a445317b"}
Sep 10 20:55:50.837261 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:50.837135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq" event={"ID":"f2cc4280-8228-453d-aa4b-98c8b86fa4cd","Type":"ContainerStarted","Data":"3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1"}
Sep 10 20:55:50.837841 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:50.837810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:55:50.871273 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:50.871232    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq" podStartSLOduration=1.571662822 podStartE2EDuration="3.871219081s" totalImagesPullingTime="2.299556259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:55:50.870472354 +0000 UTC m=+1186.781509640" watchObservedRunningTime="2026-09-10 20:55:50.871219081 +0000 UTC m=+1186.782256409"
Sep 10 20:55:53.900059 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:53.900026    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:55:56.851710 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:55:56.851661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:56:04.620739 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:04.620710    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:56:04.621202 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:04.620777    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 20:56:04.624729 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:04.624706    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:56:04.624860 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:04.624751    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 20:56:08.602635 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:08.602606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:56:24.106063 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:24.106028    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:56:44.225877 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.225845    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"]
Sep 10 20:56:44.226324 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.226074    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq" podUID="f2cc4280-8228-453d-aa4b-98c8b86fa4cd" containerName="maas-api" containerID="cri-o://3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1" gracePeriod=30
Sep 10 20:56:44.483260 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.483200    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:56:44.627798 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.627769    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-maas-api-tls\") pod \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") "
Sep 10 20:56:44.627980 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.627828    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-openshift-service-ca\") pod \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") "
Sep 10 20:56:44.627980 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.627915    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qh4m\" (UniqueName: \"kubernetes.io/projected/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-kube-api-access-6qh4m\") pod \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") "
Sep 10 20:56:44.628467 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.628168    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-metrics-certs\") pod \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\" (UID: \"f2cc4280-8228-453d-aa4b-98c8b86fa4cd\") "
Sep 10 20:56:44.628467 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.628288    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2cc4280-8228-453d-aa4b-98c8b86fa4cd" (UID: "f2cc4280-8228-453d-aa4b-98c8b86fa4cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:56:44.628636 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.628549    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:56:44.630582 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.630555    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-kube-api-access-6qh4m" (OuterVolumeSpecName: "kube-api-access-6qh4m") pod "f2cc4280-8228-453d-aa4b-98c8b86fa4cd" (UID: "f2cc4280-8228-453d-aa4b-98c8b86fa4cd"). InnerVolumeSpecName "kube-api-access-6qh4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:56:44.630732 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.630599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2cc4280-8228-453d-aa4b-98c8b86fa4cd" (UID: "f2cc4280-8228-453d-aa4b-98c8b86fa4cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:56:44.630732 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.630645    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2cc4280-8228-453d-aa4b-98c8b86fa4cd" (UID: "f2cc4280-8228-453d-aa4b-98c8b86fa4cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:56:44.729551 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.729523    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:56:44.729551 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.729547    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qh4m\" (UniqueName: \"kubernetes.io/projected/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-kube-api-access-6qh4m\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:56:44.729785 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.729558    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cc4280-8228-453d-aa4b-98c8b86fa4cd-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:56:44.924626 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.924596    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"]
Sep 10 20:56:44.925119 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.925100    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cc4280-8228-453d-aa4b-98c8b86fa4cd" containerName="maas-api"
Sep 10 20:56:44.925214 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.925121    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cc4280-8228-453d-aa4b-98c8b86fa4cd" containerName="maas-api"
Sep 10 20:56:44.925265 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.925237    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cc4280-8228-453d-aa4b-98c8b86fa4cd" containerName="maas-api"
Sep 10 20:56:44.928596 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.928574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:44.937251 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.937226    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fc6e011f-serving-cert\""
Sep 10 20:56:44.944559 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:44.944536    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"]
Sep 10 20:56:45.030982 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.030955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29af1ec0-8256-4c34-a826-8f14235d9d9e-openshift-service-ca\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.031147 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.030993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-maas-api-tls\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.031147 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.031020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-metrics-certs\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.031147 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.031128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dlm\" (UniqueName: \"kubernetes.io/projected/29af1ec0-8256-4c34-a826-8f14235d9d9e-kube-api-access-45dlm\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.132581 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.132543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dlm\" (UniqueName: \"kubernetes.io/projected/29af1ec0-8256-4c34-a826-8f14235d9d9e-kube-api-access-45dlm\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.132783 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.132618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29af1ec0-8256-4c34-a826-8f14235d9d9e-openshift-service-ca\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.132783 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.132643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-maas-api-tls\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.132783 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.132668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-metrics-certs\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.133293 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.133272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29af1ec0-8256-4c34-a826-8f14235d9d9e-openshift-service-ca\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.135145 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.135118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-maas-api-tls\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.135259 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.135238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-metrics-certs\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.147527 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.147507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dlm\" (UniqueName: \"kubernetes.io/projected/29af1ec0-8256-4c34-a826-8f14235d9d9e-kube-api-access-45dlm\") pod \"maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.160576 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.160553    2565 generic.go:356] "Generic (PLEG): container finished" podID="f2cc4280-8228-453d-aa4b-98c8b86fa4cd" containerID="3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1" exitCode=0
Sep 10 20:56:45.160670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.160592    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"
Sep 10 20:56:45.160670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.160618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq" event={"ID":"f2cc4280-8228-453d-aa4b-98c8b86fa4cd","Type":"ContainerDied","Data":"3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1"}
Sep 10 20:56:45.160670 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.160659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq" event={"ID":"f2cc4280-8228-453d-aa4b-98c8b86fa4cd","Type":"ContainerDied","Data":"0a9a2b8177e0e5390c0e8879c383324b85d22aac4f73ee0c4dcbb087a445317b"}
Sep 10 20:56:45.160818 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.160680    2565 scope.go:117] "RemoveContainer" containerID="3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1"
Sep 10 20:56:45.171089 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.171001    2565 scope.go:117] "RemoveContainer" containerID="3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1"
Sep 10 20:56:45.171589 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:56:45.171518    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1\": container with ID starting with 3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1 not found: ID does not exist" containerID="3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1"
Sep 10 20:56:45.171589 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.171552    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1"} err="failed to get container status \"3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1\": rpc error: code = NotFound desc = could not find container \"3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1\": container with ID starting with 3670355467d43fcd133612fe2e976539417c45d881f32655c85855e2a98d92c1 not found: ID does not exist"
Sep 10 20:56:45.196411 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.196356    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"]
Sep 10 20:56:45.212141 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.212120    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75dcbb-85d4f7c86-q8tvq"]
Sep 10 20:56:45.241380 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.241359    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:45.514254 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:45.514233    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"]
Sep 10 20:56:45.515836 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:56:45.515808    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29af1ec0_8256_4c34_a826_8f14235d9d9e.slice/crio-32af420e7664628cf62fbe9bb83f477d7261e08a10925da06cce5c3ab80718ae WatchSource:0}: Error finding container 32af420e7664628cf62fbe9bb83f477d7261e08a10925da06cce5c3ab80718ae: Status 404 returned error can't find the container with id 32af420e7664628cf62fbe9bb83f477d7261e08a10925da06cce5c3ab80718ae
Sep 10 20:56:46.169013 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:46.168921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh" event={"ID":"29af1ec0-8256-4c34-a826-8f14235d9d9e","Type":"ContainerStarted","Data":"28e8cf7fe95c9315ea2e3a4fd00c80a4635c16a1256f86823b90687aa2ab3b8c"}
Sep 10 20:56:46.169013 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:46.168982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh" event={"ID":"29af1ec0-8256-4c34-a826-8f14235d9d9e","Type":"ContainerStarted","Data":"32af420e7664628cf62fbe9bb83f477d7261e08a10925da06cce5c3ab80718ae"}
Sep 10 20:56:46.169222 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:46.169129    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:56:46.190567 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:46.190521    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh" podStartSLOduration=1.838370922 podStartE2EDuration="2.190505364s" totalImagesPullingTime="352.134442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:56:46.189049368 +0000 UTC m=+1242.100086653" watchObservedRunningTime="2026-09-10 20:56:46.190505364 +0000 UTC m=+1242.101542647"
Sep 10 20:56:46.592058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:46.592028    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cc4280-8228-453d-aa4b-98c8b86fa4cd" path="/var/lib/kubelet/pods/f2cc4280-8228-453d-aa4b-98c8b86fa4cd/volumes"
Sep 10 20:56:48.131117 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:48.131082    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:56:52.181544 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:56:52.181515    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:57:06.221072 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:06.221036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:57:28.281101 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.281061    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"]
Sep 10 20:57:28.281624 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.281294    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh" podUID="29af1ec0-8256-4c34-a826-8f14235d9d9e" containerName="maas-api" containerID="cri-o://28e8cf7fe95c9315ea2e3a4fd00c80a4635c16a1256f86823b90687aa2ab3b8c" gracePeriod=30
Sep 10 20:57:28.413261 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.413232    2565 generic.go:356] "Generic (PLEG): container finished" podID="29af1ec0-8256-4c34-a826-8f14235d9d9e" containerID="28e8cf7fe95c9315ea2e3a4fd00c80a4635c16a1256f86823b90687aa2ab3b8c" exitCode=0
Sep 10 20:57:28.413414 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.413308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh" event={"ID":"29af1ec0-8256-4c34-a826-8f14235d9d9e","Type":"ContainerDied","Data":"28e8cf7fe95c9315ea2e3a4fd00c80a4635c16a1256f86823b90687aa2ab3b8c"}
Sep 10 20:57:28.542445 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.542381    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:57:28.703044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.703015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-metrics-certs\") pod \"29af1ec0-8256-4c34-a826-8f14235d9d9e\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") "
Sep 10 20:57:28.703044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.703048    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-maas-api-tls\") pod \"29af1ec0-8256-4c34-a826-8f14235d9d9e\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") "
Sep 10 20:57:28.703281 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.703259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29af1ec0-8256-4c34-a826-8f14235d9d9e-openshift-service-ca\") pod \"29af1ec0-8256-4c34-a826-8f14235d9d9e\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") "
Sep 10 20:57:28.703347 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.703307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45dlm\" (UniqueName: \"kubernetes.io/projected/29af1ec0-8256-4c34-a826-8f14235d9d9e-kube-api-access-45dlm\") pod \"29af1ec0-8256-4c34-a826-8f14235d9d9e\" (UID: \"29af1ec0-8256-4c34-a826-8f14235d9d9e\") "
Sep 10 20:57:28.703616 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.703576    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29af1ec0-8256-4c34-a826-8f14235d9d9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29af1ec0-8256-4c34-a826-8f14235d9d9e" (UID: "29af1ec0-8256-4c34-a826-8f14235d9d9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:28.703760 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.703748    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29af1ec0-8256-4c34-a826-8f14235d9d9e-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:28.705148 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.705122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29af1ec0-8256-4c34-a826-8f14235d9d9e" (UID: "29af1ec0-8256-4c34-a826-8f14235d9d9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:57:28.705236 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.705159    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29af1ec0-8256-4c34-a826-8f14235d9d9e" (UID: "29af1ec0-8256-4c34-a826-8f14235d9d9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:57:28.705236 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.705193    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29af1ec0-8256-4c34-a826-8f14235d9d9e-kube-api-access-45dlm" (OuterVolumeSpecName: "kube-api-access-45dlm") pod "29af1ec0-8256-4c34-a826-8f14235d9d9e" (UID: "29af1ec0-8256-4c34-a826-8f14235d9d9e"). InnerVolumeSpecName "kube-api-access-45dlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:28.804663 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.804602    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:28.804663 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.804626    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45dlm\" (UniqueName: \"kubernetes.io/projected/29af1ec0-8256-4c34-a826-8f14235d9d9e-kube-api-access-45dlm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:28.804663 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:28.804637    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29af1ec0-8256-4c34-a826-8f14235d9d9e-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:29.420802 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:29.420768    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"
Sep 10 20:57:29.421246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:29.420814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh" event={"ID":"29af1ec0-8256-4c34-a826-8f14235d9d9e","Type":"ContainerDied","Data":"32af420e7664628cf62fbe9bb83f477d7261e08a10925da06cce5c3ab80718ae"}
Sep 10 20:57:29.421246 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:29.420877    2565 scope.go:117] "RemoveContainer" containerID="28e8cf7fe95c9315ea2e3a4fd00c80a4635c16a1256f86823b90687aa2ab3b8c"
Sep 10 20:57:29.445892 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:29.445849    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"]
Sep 10 20:57:29.448556 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:29.448535    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc6e011f-7d58dc9f9d-4cqkh"]
Sep 10 20:57:30.591572 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:30.591537    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29af1ec0-8256-4c34-a826-8f14235d9d9e" path="/var/lib/kubelet/pods/29af1ec0-8256-4c34-a826-8f14235d9d9e/volumes"
Sep 10 20:57:32.003108 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.003078    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"]
Sep 10 20:57:32.003519 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.003506    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29af1ec0-8256-4c34-a826-8f14235d9d9e" containerName="maas-api"
Sep 10 20:57:32.003519 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.003519    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29af1ec0-8256-4c34-a826-8f14235d9d9e" containerName="maas-api"
Sep 10 20:57:32.003593 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.003584    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29af1ec0-8256-4c34-a826-8f14235d9d9e" containerName="maas-api"
Sep 10 20:57:32.008051 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.008033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.013056 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.013035    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3671a4\"/\"default-dockercfg-sgllx\""
Sep 10 20:57:32.013180 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.013080    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3671a4\"/\"noten-model-3671a4-kserve-self-signed-certs\""
Sep 10 20:57:32.013180 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.013082    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3671a4\"/\"kube-root-ca.crt\""
Sep 10 20:57:32.013180 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.013088    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3671a4\"/\"openshift-service-ca.crt\""
Sep 10 20:57:32.018262 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.018240    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"]
Sep 10 20:57:32.132965 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.132935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-dshm\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.133122 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.132982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.133122 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.133024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-model-cache\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.133122 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.133054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tmp-dir\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.133345 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.133155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-home\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.133345 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.133226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjn9m\" (UniqueName: \"kubernetes.io/projected/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-kube-api-access-sjn9m\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.233925 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.233893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-model-cache\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234065 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.233941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tmp-dir\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234065 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.233981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-home\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234065 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.234024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjn9m\" (UniqueName: \"kubernetes.io/projected/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-kube-api-access-sjn9m\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234235 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.234089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-dshm\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234235 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.234139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234350 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.234331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-model-cache\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234413 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.234382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tmp-dir\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.234675 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.234647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-home\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.236499 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.236478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-dshm\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.236671 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.236653    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.244286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.244264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjn9m\" (UniqueName: \"kubernetes.io/projected/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-kube-api-access-sjn9m\") pod \"noten-model-3671a4-kserve-59dfb76467-954gl\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") " pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.317642 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.317567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:32.593976 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:32.593946    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"]
Sep 10 20:57:32.595015 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:57:32.594992    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ce67ef5_72f6_47a4_b85f_0e94a0b82511.slice/crio-37bee2e8d8f3a0958f12191c42f0e671ed9176081e75a74b0c5bc1f216f3a2cd WatchSource:0}: Error finding container 37bee2e8d8f3a0958f12191c42f0e671ed9176081e75a74b0c5bc1f216f3a2cd: Status 404 returned error can't find the container with id 37bee2e8d8f3a0958f12191c42f0e671ed9176081e75a74b0c5bc1f216f3a2cd
Sep 10 20:57:33.446803 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:33.446768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl" event={"ID":"3ce67ef5-72f6-47a4-b85f-0e94a0b82511","Type":"ContainerStarted","Data":"01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f"}
Sep 10 20:57:33.447194 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:33.446817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl" event={"ID":"3ce67ef5-72f6-47a4-b85f-0e94a0b82511","Type":"ContainerStarted","Data":"37bee2e8d8f3a0958f12191c42f0e671ed9176081e75a74b0c5bc1f216f3a2cd"}
Sep 10 20:57:33.479079 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:33.479027    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl" podStartSLOduration=2.47901188 podStartE2EDuration="2.47901188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:33.477383011 +0000 UTC m=+1289.388420296" watchObservedRunningTime="2026-09-10 20:57:33.47901188 +0000 UTC m=+1289.390049198"
Sep 10 20:57:34.047345 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:34.047312    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:57:34.825156 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:34.825123    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"]
Sep 10 20:57:34.828650 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:34.828632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"
Sep 10 20:57:34.832374 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:34.832349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:57:34.832374 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:34.832350    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:57:34.850755 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:34.850728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"]
Sep 10 20:57:34.960332 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:34.960297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh\" (UID: \"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"
Sep 10 20:57:35.061392 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:35.061361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh\" (UID: \"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"
Sep 10 20:57:35.062049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:35.062022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh\" (UID: \"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"
Sep 10 20:57:35.140488 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:35.140459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"
Sep 10 20:57:35.424709 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:35.424652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"]
Sep 10 20:57:35.425447 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:57:35.425419    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ff6e92_8e8b_4c91_b632_6aa2cbf6fb49.slice/crio-4e20b9cc129008adf4eeb1bc10765775419873a9436cb7f724623d2347c5f9b5 WatchSource:0}: Error finding container 4e20b9cc129008adf4eeb1bc10765775419873a9436cb7f724623d2347c5f9b5: Status 404 returned error can't find the container with id 4e20b9cc129008adf4eeb1bc10765775419873a9436cb7f724623d2347c5f9b5
Sep 10 20:57:35.459179 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:35.459156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh" event={"ID":"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49","Type":"ContainerStarted","Data":"4e20b9cc129008adf4eeb1bc10765775419873a9436cb7f724623d2347c5f9b5"}
Sep 10 20:57:36.466383 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:36.466350    2565 generic.go:356] "Generic (PLEG): container finished" podID="98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49" containerID="c11c8d14509cf7bb496a01be18c059910c372951ed08e3d9dfa93a78de579a38" exitCode=0
Sep 10 20:57:36.466765 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:36.466440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh" event={"ID":"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49","Type":"ContainerDied","Data":"c11c8d14509cf7bb496a01be18c059910c372951ed08e3d9dfa93a78de579a38"}
Sep 10 20:57:37.612507 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:37.612483    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"
Sep 10 20:57:37.787504 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:37.787411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49-maas-api-service-ca\") pod \"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49\" (UID: \"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49\") "
Sep 10 20:57:37.787774 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:37.787751    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49" (UID: "98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:37.888594 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:37.888564    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:38.478954 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:38.478920    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"
Sep 10 20:57:38.479121 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:38.478959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh" event={"ID":"98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49","Type":"ContainerDied","Data":"4e20b9cc129008adf4eeb1bc10765775419873a9436cb7f724623d2347c5f9b5"}
Sep 10 20:57:38.479121 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:38.479004    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e20b9cc129008adf4eeb1bc10765775419873a9436cb7f724623d2347c5f9b5"
Sep 10 20:57:42.318298 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:42.318260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:42.318757 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:42.318319    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:42.331015 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:42.330990    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:42.512513 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:42.512485    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:43.519800 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:43.519767    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"]
Sep 10 20:57:43.539120 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:43.539094    2565 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-self-signed-certs: secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:43.539241 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:43.539158    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs podName:3ce67ef5-72f6-47a4-b85f-0e94a0b82511 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:44.039142459 +0000 UTC m=+1299.950179734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs") pod "noten-model-3671a4-kserve-59dfb76467-954gl" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511") : secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:44.042833 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:44.042796    2565 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-self-signed-certs: secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:44.042999 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:44.042879    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs podName:3ce67ef5-72f6-47a4-b85f-0e94a0b82511 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.042859936 +0000 UTC m=+1300.953897220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs") pod "noten-model-3671a4-kserve-59dfb76467-954gl" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511") : secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:45.051080 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:45.051044    2565 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-self-signed-certs: secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:45.051537 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:45.051125    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs podName:3ce67ef5-72f6-47a4-b85f-0e94a0b82511 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:47.051107359 +0000 UTC m=+1302.962144621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs") pod "noten-model-3671a4-kserve-59dfb76467-954gl" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511") : secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:46.172010 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.171974    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"]
Sep 10 20:57:46.172412 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.172212    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk" podUID="db3c7d77-b4a0-4f87-9856-1e1a618db392" containerName="payload-processing" containerID="cri-o://43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271" gracePeriod=30
Sep 10 20:57:46.445851 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.445824    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:57:46.465205 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.465173    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3c7d77-b4a0-4f87-9856-1e1a618db392-plugins-config-volume\") pod \"db3c7d77-b4a0-4f87-9856-1e1a618db392\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") "
Sep 10 20:57:46.465342 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.465264    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzs9n\" (UniqueName: \"kubernetes.io/projected/db3c7d77-b4a0-4f87-9856-1e1a618db392-kube-api-access-mzs9n\") pod \"db3c7d77-b4a0-4f87-9856-1e1a618db392\" (UID: \"db3c7d77-b4a0-4f87-9856-1e1a618db392\") "
Sep 10 20:57:46.465636 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.465608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3c7d77-b4a0-4f87-9856-1e1a618db392-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db3c7d77-b4a0-4f87-9856-1e1a618db392" (UID: "db3c7d77-b4a0-4f87-9856-1e1a618db392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:46.467633 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.467604    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3c7d77-b4a0-4f87-9856-1e1a618db392-kube-api-access-mzs9n" (OuterVolumeSpecName: "kube-api-access-mzs9n") pod "db3c7d77-b4a0-4f87-9856-1e1a618db392" (UID: "db3c7d77-b4a0-4f87-9856-1e1a618db392"). InnerVolumeSpecName "kube-api-access-mzs9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:46.526059 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.526032    2565 generic.go:356] "Generic (PLEG): container finished" podID="db3c7d77-b4a0-4f87-9856-1e1a618db392" containerID="43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271" exitCode=0
Sep 10 20:57:46.526197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.526072    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"
Sep 10 20:57:46.526197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.526081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk" event={"ID":"db3c7d77-b4a0-4f87-9856-1e1a618db392","Type":"ContainerDied","Data":"43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271"}
Sep 10 20:57:46.526197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.526137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk" event={"ID":"db3c7d77-b4a0-4f87-9856-1e1a618db392","Type":"ContainerDied","Data":"f550a2c582dd997ff9343af8c653031d07c5b5cf0ad7579807481d3a8000d104"}
Sep 10 20:57:46.526197 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.526168    2565 scope.go:117] "RemoveContainer" containerID="43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271"
Sep 10 20:57:46.537714 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.537673    2565 scope.go:117] "RemoveContainer" containerID="43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271"
Sep 10 20:57:46.541337 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:46.541309    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271\": container with ID starting with 43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271 not found: ID does not exist" containerID="43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271"
Sep 10 20:57:46.541451 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.541338    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271"} err="failed to get container status \"43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271\": rpc error: code = NotFound desc = could not find container \"43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271\": container with ID starting with 43629f03dd656975c2f5b5da9d43c68ecef831a89c1885dd39d78bfaa62f5271 not found: ID does not exist"
Sep 10 20:57:46.565010 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.564990    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"]
Sep 10 20:57:46.567072 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.567054    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3c7d77-b4a0-4f87-9856-1e1a618db392-plugins-config-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:46.567130 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.567075    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzs9n\" (UniqueName: \"kubernetes.io/projected/db3c7d77-b4a0-4f87-9856-1e1a618db392-kube-api-access-mzs9n\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:46.571199 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.571179    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"]
Sep 10 20:57:46.573186 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.573158    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80aa25-vh9lh"]
Sep 10 20:57:46.582256 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.576281    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80aa25-d8499f755-557zk"]
Sep 10 20:57:46.592136 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.592110    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49" path="/var/lib/kubelet/pods/98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49/volumes"
Sep 10 20:57:46.592529 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.592515    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3c7d77-b4a0-4f87-9856-1e1a618db392" path="/var/lib/kubelet/pods/db3c7d77-b4a0-4f87-9856-1e1a618db392/volumes"
Sep 10 20:57:46.750239 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:46.750156    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:57:47.073496 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:47.073426    2565 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-self-signed-certs: secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:47.073496 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:47.073497    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs podName:3ce67ef5-72f6-47a4-b85f-0e94a0b82511 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:51.073482148 +0000 UTC m=+1306.984519423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs") pod "noten-model-3671a4-kserve-59dfb76467-954gl" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511") : secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:51.111915 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:51.111887    2565 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-self-signed-certs: secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:51.112301 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:51.111958    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs podName:3ce67ef5-72f6-47a4-b85f-0e94a0b82511 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:59.111943863 +0000 UTC m=+1315.022981125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs") pod "noten-model-3671a4-kserve-59dfb76467-954gl" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511") : secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:54.631828 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.631799    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"]
Sep 10 20:57:54.632214 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.632201    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49" containerName="revoke-keys"
Sep 10 20:57:54.632214 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.632214    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49" containerName="revoke-keys"
Sep 10 20:57:54.632295 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.632232    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3c7d77-b4a0-4f87-9856-1e1a618db392" containerName="payload-processing"
Sep 10 20:57:54.632295 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.632238    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3c7d77-b4a0-4f87-9856-1e1a618db392" containerName="payload-processing"
Sep 10 20:57:54.632369 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.632312    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ff6e92-8e8b-4c91-b632-6aa2cbf6fb49" containerName="revoke-keys"
Sep 10 20:57:54.632369 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.632322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3c7d77-b4a0-4f87-9856-1e1a618db392" containerName="payload-processing"
Sep 10 20:57:54.636770 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.636750    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"
Sep 10 20:57:54.640009 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.639990    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:57:54.640167 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.639993    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.644945 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.644815    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"]
Sep 10 20:57:54.746541 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.746513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2199f176-1e30-4da4-9877-6b99deb6078f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm\" (UID: \"2199f176-1e30-4da4-9877-6b99deb6078f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"
Sep 10 20:57:54.847855 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.847831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2199f176-1e30-4da4-9877-6b99deb6078f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm\" (UID: \"2199f176-1e30-4da4-9877-6b99deb6078f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"
Sep 10 20:57:54.848399 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.848378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2199f176-1e30-4da4-9877-6b99deb6078f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm\" (UID: \"2199f176-1e30-4da4-9877-6b99deb6078f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"
Sep 10 20:57:54.950204 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:54.950123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"
Sep 10 20:57:55.223496 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:55.223418    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"]
Sep 10 20:57:55.226233 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:57:55.226207    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2199f176_1e30_4da4_9877_6b99deb6078f.slice/crio-f06060919b77eca48409c5f298e5a575f3d78a8193c344b46d362a04322468ec WatchSource:0}: Error finding container f06060919b77eca48409c5f298e5a575f3d78a8193c344b46d362a04322468ec: Status 404 returned error can't find the container with id f06060919b77eca48409c5f298e5a575f3d78a8193c344b46d362a04322468ec
Sep 10 20:57:55.582814 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:55.582780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm" event={"ID":"2199f176-1e30-4da4-9877-6b99deb6078f","Type":"ContainerStarted","Data":"0061dd21a042cd5138c5fb897d8b2bd6e01ea9d1c5f8abb6bbd146629055d305"}
Sep 10 20:57:55.582928 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:55.582848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm" event={"ID":"2199f176-1e30-4da4-9877-6b99deb6078f","Type":"ContainerStarted","Data":"f06060919b77eca48409c5f298e5a575f3d78a8193c344b46d362a04322468ec"}
Sep 10 20:57:55.605097 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:55.605055    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm" podStartSLOduration=1.605039964 podStartE2EDuration="1.605039964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:55.603240177 +0000 UTC m=+1311.514277472" watchObservedRunningTime="2026-09-10 20:57:55.605039964 +0000 UTC m=+1311.516077248"
Sep 10 20:57:56.526044 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:56.526009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:57:56.590326 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:56.590296    2565 generic.go:356] "Generic (PLEG): container finished" podID="2199f176-1e30-4da4-9877-6b99deb6078f" containerID="0061dd21a042cd5138c5fb897d8b2bd6e01ea9d1c5f8abb6bbd146629055d305" exitCode=0
Sep 10 20:57:56.591253 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:56.591231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm" event={"ID":"2199f176-1e30-4da4-9877-6b99deb6078f","Type":"ContainerDied","Data":"0061dd21a042cd5138c5fb897d8b2bd6e01ea9d1c5f8abb6bbd146629055d305"}
Sep 10 20:57:57.721135 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:57.721107    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"
Sep 10 20:57:57.772620 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:57.772592    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2199f176-1e30-4da4-9877-6b99deb6078f-maas-api-service-ca\") pod \"2199f176-1e30-4da4-9877-6b99deb6078f\" (UID: \"2199f176-1e30-4da4-9877-6b99deb6078f\") "
Sep 10 20:57:57.772938 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:57.772913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2199f176-1e30-4da4-9877-6b99deb6078f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2199f176-1e30-4da4-9877-6b99deb6078f" (UID: "2199f176-1e30-4da4-9877-6b99deb6078f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:57.873543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:57.873477    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2199f176-1e30-4da4-9877-6b99deb6078f-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:58.602428 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:58.602405    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"
Sep 10 20:57:58.602543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:58.602400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm" event={"ID":"2199f176-1e30-4da4-9877-6b99deb6078f","Type":"ContainerDied","Data":"f06060919b77eca48409c5f298e5a575f3d78a8193c344b46d362a04322468ec"}
Sep 10 20:57:58.602543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:58.602526    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f06060919b77eca48409c5f298e5a575f3d78a8193c344b46d362a04322468ec"
Sep 10 20:57:59.184537 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:59.184505    2565 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-self-signed-certs: secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:59.185093 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:57:59.184588    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs podName:3ce67ef5-72f6-47a4-b85f-0e94a0b82511 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:15.18456901 +0000 UTC m=+1331.095606272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs") pod "noten-model-3671a4-kserve-59dfb76467-954gl" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511") : secret "noten-model-3671a4-kserve-self-signed-certs" not found
Sep 10 20:57:59.532367 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.532285    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl" podUID="3ce67ef5-72f6-47a4-b85f-0e94a0b82511" containerName="main" containerID="cri-o://01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f" gracePeriod=45
Sep 10 20:57:59.794368 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.794314    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:57:59.890150 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890122    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tmp-dir\") pod \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") "
Sep 10 20:57:59.890287 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-home\") pod \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") "
Sep 10 20:57:59.890287 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890209    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs\") pod \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") "
Sep 10 20:57:59.890287 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-dshm\") pod \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") "
Sep 10 20:57:59.890287 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890247    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjn9m\" (UniqueName: \"kubernetes.io/projected/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-kube-api-access-sjn9m\") pod \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") "
Sep 10 20:57:59.890287 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-model-cache\") pod \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\" (UID: \"3ce67ef5-72f6-47a4-b85f-0e94a0b82511\") "
Sep 10 20:57:59.890534 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ce67ef5-72f6-47a4-b85f-0e94a0b82511" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.890595 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890541    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-home" (OuterVolumeSpecName: "home") pod "3ce67ef5-72f6-47a4-b85f-0e94a0b82511" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.890652 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890589    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.890652 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.890625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ce67ef5-72f6-47a4-b85f-0e94a0b82511" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.892400 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.892381    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-dshm" (OuterVolumeSpecName: "dshm") pod "3ce67ef5-72f6-47a4-b85f-0e94a0b82511" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.892468 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.892414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ce67ef5-72f6-47a4-b85f-0e94a0b82511" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:57:59.892517 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.892499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-kube-api-access-sjn9m" (OuterVolumeSpecName: "kube-api-access-sjn9m") pod "3ce67ef5-72f6-47a4-b85f-0e94a0b82511" (UID: "3ce67ef5-72f6-47a4-b85f-0e94a0b82511"). InnerVolumeSpecName "kube-api-access-sjn9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:59.991192 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.991169    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.991192 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.991189    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.991333 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.991198    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjn9m\" (UniqueName: \"kubernetes.io/projected/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-kube-api-access-sjn9m\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.991333 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.991208    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.991333 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:57:59.991216    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce67ef5-72f6-47a4-b85f-0e94a0b82511-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:00.398870 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.398835    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"]
Sep 10 20:58:00.399316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.399247    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce67ef5-72f6-47a4-b85f-0e94a0b82511" containerName="main"
Sep 10 20:58:00.399316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.399258    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce67ef5-72f6-47a4-b85f-0e94a0b82511" containerName="main"
Sep 10 20:58:00.399316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.399268    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2199f176-1e30-4da4-9877-6b99deb6078f" containerName="revoke-keys"
Sep 10 20:58:00.399316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.399274    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2199f176-1e30-4da4-9877-6b99deb6078f" containerName="revoke-keys"
Sep 10 20:58:00.399549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.399338    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce67ef5-72f6-47a4-b85f-0e94a0b82511" containerName="main"
Sep 10 20:58:00.399549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.399349    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2199f176-1e30-4da4-9877-6b99deb6078f" containerName="revoke-keys"
Sep 10 20:58:00.402701 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.402670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.407429 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.407392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-cd89ac-serving-cert\""
Sep 10 20:58:00.407429 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.407406    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:58:00.407637 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.407459    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 20:58:00.407869 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.407738    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 20:58:00.409154 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.409135    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqzh2\""
Sep 10 20:58:00.422861 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.422826    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"]
Sep 10 20:58:00.496573 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.496545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.496761 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.496595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.496761 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.496653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svtt\" (UniqueName: \"kubernetes.io/projected/78f228c4-8364-46ef-93d1-78c5c4293ee2-kube-api-access-9svtt\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.496761 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.496715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f228c4-8364-46ef-93d1-78c5c4293ee2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.597718 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.597672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.597884 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.597763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svtt\" (UniqueName: \"kubernetes.io/projected/78f228c4-8364-46ef-93d1-78c5c4293ee2-kube-api-access-9svtt\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.597884 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.597790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f228c4-8364-46ef-93d1-78c5c4293ee2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.598297 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.597921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.598541 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.598522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f228c4-8364-46ef-93d1-78c5c4293ee2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.600163 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.600137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.600291 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.600270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.612493 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.612473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svtt\" (UniqueName: \"kubernetes.io/projected/78f228c4-8364-46ef-93d1-78c5c4293ee2-kube-api-access-9svtt\") pod \"maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.615457 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.615433    2565 generic.go:356] "Generic (PLEG): container finished" podID="3ce67ef5-72f6-47a4-b85f-0e94a0b82511" containerID="01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f" exitCode=0
Sep 10 20:58:00.615560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.615468    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"
Sep 10 20:58:00.615560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.615511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl" event={"ID":"3ce67ef5-72f6-47a4-b85f-0e94a0b82511","Type":"ContainerDied","Data":"01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f"}
Sep 10 20:58:00.615560 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.615545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl" event={"ID":"3ce67ef5-72f6-47a4-b85f-0e94a0b82511","Type":"ContainerDied","Data":"37bee2e8d8f3a0958f12191c42f0e671ed9176081e75a74b0c5bc1f216f3a2cd"}
Sep 10 20:58:00.615810 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.615563    2565 scope.go:117] "RemoveContainer" containerID="01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f"
Sep 10 20:58:00.628058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.628032    2565 scope.go:117] "RemoveContainer" containerID="01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f"
Sep 10 20:58:00.628355 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:00.628332    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f\": container with ID starting with 01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f not found: ID does not exist" containerID="01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f"
Sep 10 20:58:00.628414 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.628364    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f"} err="failed to get container status \"01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f\": rpc error: code = NotFound desc = could not find container \"01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f\": container with ID starting with 01b592311c7c0c00811dff6be80df51f99a85a1682627481c477463a854ad41f not found: ID does not exist"
Sep 10 20:58:00.643633 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.643610    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"]
Sep 10 20:58:00.656926 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.656875    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-3671a4/noten-model-3671a4-kserve-59dfb76467-954gl"]
Sep 10 20:58:00.714140 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.714121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:00.996769 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:00.995503    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"]
Sep 10 20:58:00.999134 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:58:00.999083    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f228c4_8364_46ef_93d1_78c5c4293ee2.slice/crio-f626ea317c4c59faa7e058bc0602e0fc03842040d84702e4c2679c6899b670d1 WatchSource:0}: Error finding container f626ea317c4c59faa7e058bc0602e0fc03842040d84702e4c2679c6899b670d1: Status 404 returned error can't find the container with id f626ea317c4c59faa7e058bc0602e0fc03842040d84702e4c2679c6899b670d1
Sep 10 20:58:01.623606 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:01.623575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9" event={"ID":"78f228c4-8364-46ef-93d1-78c5c4293ee2","Type":"ContainerStarted","Data":"08aaacba053f30bb53bb309a265dd5653c624ae71af337405451f2deafeb1d98"}
Sep 10 20:58:01.623957 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:01.623619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9" event={"ID":"78f228c4-8364-46ef-93d1-78c5c4293ee2","Type":"ContainerStarted","Data":"f626ea317c4c59faa7e058bc0602e0fc03842040d84702e4c2679c6899b670d1"}
Sep 10 20:58:01.623957 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:01.623658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:01.646286 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:01.646236    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9" podStartSLOduration=1.192837506 podStartE2EDuration="1.646220996s" totalImagesPullingTime="453.38349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:01.645794946 +0000 UTC m=+1317.556832233" watchObservedRunningTime="2026-09-10 20:58:01.646220996 +0000 UTC m=+1317.557258280"
Sep 10 20:58:02.591147 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:02.591109    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce67ef5-72f6-47a4-b85f-0e94a0b82511" path="/var/lib/kubelet/pods/3ce67ef5-72f6-47a4-b85f-0e94a0b82511/volumes"
Sep 10 20:58:04.941240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:04.941198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:58:06.398382 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:06.398348    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"]
Sep 10 20:58:06.405870 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:06.405847    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-467fc3-ftrdm"]
Sep 10 20:58:06.591608 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:06.591575    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2199f176-1e30-4da4-9877-6b99deb6078f" path="/var/lib/kubelet/pods/2199f176-1e30-4da4-9877-6b99deb6078f/volumes"
Sep 10 20:58:07.633829 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:07.633791    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 20:58:12.221090 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:12.221054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:58:13.897965 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:13.897932    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"]
Sep 10 20:58:13.898341 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:13.898141    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy" containerID="cri-o://103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8" gracePeriod=30
Sep 10 20:58:19.154599 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.154574    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:58:19.261130 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-podinfo\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261130 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261101    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-envoy\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261130 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istiod-ca-cert\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261161    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-credential-socket\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261213    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-data\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-certs\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-socket\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261344    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sj9h\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-kube-api-access-2sj9h\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261384 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261368    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-token\") pod \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\" (UID: \"2565bb7c-bc0d-4d2a-9bfc-756d671f9373\") "
Sep 10 20:58:19.261735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261523    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-data" (OuterVolumeSpecName: "istio-data") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:58:19.261735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:58:19.261735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261560    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:58:19.261735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:58:19.261735 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261730    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:58:19.262007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261773    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-data\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.262007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261793    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.262007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261806    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istiod-ca-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.262007 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.261819    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-credential-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.263345 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.263319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:58:19.263445 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.263418    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 20:58:19.263530 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.263511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-token" (OuterVolumeSpecName: "istio-token") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:58:19.263566 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.263533    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-kube-api-access-2sj9h" (OuterVolumeSpecName: "kube-api-access-2sj9h") pod "2565bb7c-bc0d-4d2a-9bfc-756d671f9373" (UID: "2565bb7c-bc0d-4d2a-9bfc-756d671f9373"). InnerVolumeSpecName "kube-api-access-2sj9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:58:19.363050 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.363029    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-podinfo\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.363050 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.363051    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-envoy\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.363181 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.363061    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-workload-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.363181 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.363069    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sj9h\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-kube-api-access-2sj9h\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.363181 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.363078    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2565bb7c-bc0d-4d2a-9bfc-756d671f9373-istio-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:19.729854 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.729820    2565 generic.go:356] "Generic (PLEG): container finished" podID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerID="103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8" exitCode=0
Sep 10 20:58:19.730016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.729867    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"
Sep 10 20:58:19.730016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.729943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" event={"ID":"2565bb7c-bc0d-4d2a-9bfc-756d671f9373","Type":"ContainerDied","Data":"103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8"}
Sep 10 20:58:19.730016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.729992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr" event={"ID":"2565bb7c-bc0d-4d2a-9bfc-756d671f9373","Type":"ContainerDied","Data":"33c8af768da484e26550e3b8ff0651f75af3ae29916cfda1ef9998be0e13e706"}
Sep 10 20:58:19.730016 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.730013    2565 scope.go:117] "RemoveContainer" containerID="103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8"
Sep 10 20:58:19.741374 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.741346    2565 scope.go:117] "RemoveContainer" containerID="3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a"
Sep 10 20:58:19.749896 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.749879    2565 scope.go:117] "RemoveContainer" containerID="103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8"
Sep 10 20:58:19.750213 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:19.750185    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8\": container with ID starting with 103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8 not found: ID does not exist" containerID="103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8"
Sep 10 20:58:19.750327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.750224    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8"} err="failed to get container status \"103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8\": rpc error: code = NotFound desc = could not find container \"103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8\": container with ID starting with 103c7e039f594c1e171732bc540717a039adbdcb3c696d1a29cb64f1c6b6c6f8 not found: ID does not exist"
Sep 10 20:58:19.750327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.750262    2565 scope.go:117] "RemoveContainer" containerID="3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a"
Sep 10 20:58:19.750544 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:19.750523    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a\": container with ID starting with 3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a not found: ID does not exist" containerID="3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a"
Sep 10 20:58:19.750638 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.750548    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a"} err="failed to get container status \"3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a\": rpc error: code = NotFound desc = could not find container \"3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a\": container with ID starting with 3bac5a33c600f69a6fa679a5d1b213d2b3aaacc0f372211086edad00b6d2366a not found: ID does not exist"
Sep 10 20:58:19.758872 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.758852    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"]
Sep 10 20:58:19.766633 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.766613    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-467fc3-openshift-default-68b6c78c5d-vnpnr"]
Sep 10 20:58:19.795074 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.795055    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"]
Sep 10 20:58:19.795465 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.795454    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy"
Sep 10 20:58:19.795517 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.795467    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy"
Sep 10 20:58:19.795517 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.795477    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy"
Sep 10 20:58:19.795517 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.795482    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy"
Sep 10 20:58:19.795614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.795546    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy"
Sep 10 20:58:19.795614 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.795556    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" containerName="istio-proxy"
Sep 10 20:58:19.800030 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.800013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:19.803669 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.803649    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3fc2b8-serving-cert\""
Sep 10 20:58:19.810599 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.810576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"]
Sep 10 20:58:19.968879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.968847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:19.968879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.968882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvch\" (UniqueName: \"kubernetes.io/projected/2c228527-cca0-4b08-a949-a138f8027dc1-kube-api-access-qhvch\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:19.969070 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.968962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:19.969070 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:19.968984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c228527-cca0-4b08-a949-a138f8027dc1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.070088 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.070017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.070088 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.070056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c228527-cca0-4b08-a949-a138f8027dc1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.070252 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.070124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.070252 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.070159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvch\" (UniqueName: \"kubernetes.io/projected/2c228527-cca0-4b08-a949-a138f8027dc1-kube-api-access-qhvch\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.070252 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:20.070177    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-serving-cert: secret "maas-api-e2e-shared-a-w3-3fc2b8-serving-cert" not found
Sep 10 20:58:20.070252 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:20.070237    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls podName:2c228527-cca0-4b08-a949-a138f8027dc1 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:20.570218413 +0000 UTC m=+1336.481255688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls") pod "maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw" (UID: "2c228527-cca0-4b08-a949-a138f8027dc1") : secret "maas-api-e2e-shared-a-w3-3fc2b8-serving-cert" not found
Sep 10 20:58:20.070725 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.070699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c228527-cca0-4b08-a949-a138f8027dc1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.072604 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.072579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.082117 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.082096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvch\" (UniqueName: \"kubernetes.io/projected/2c228527-cca0-4b08-a949-a138f8027dc1-kube-api-access-qhvch\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.575190 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.575152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.577483 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.577460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.591372 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.591343    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2565bb7c-bc0d-4d2a-9bfc-756d671f9373" path="/var/lib/kubelet/pods/2565bb7c-bc0d-4d2a-9bfc-756d671f9373/volumes"
Sep 10 20:58:20.714772 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.714743    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:20.992813 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:20.992787    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"]
Sep 10 20:58:20.993556 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:58:20.993530    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c228527_cca0_4b08_a949_a138f8027dc1.slice/crio-a98c85681aac5777cbb69e2c25b1f70578ff76804be4ce133dfe25b482ee4d71 WatchSource:0}: Error finding container a98c85681aac5777cbb69e2c25b1f70578ff76804be4ce133dfe25b482ee4d71: Status 404 returned error can't find the container with id a98c85681aac5777cbb69e2c25b1f70578ff76804be4ce133dfe25b482ee4d71
Sep 10 20:58:21.748712 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:21.748652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw" event={"ID":"2c228527-cca0-4b08-a949-a138f8027dc1","Type":"ContainerStarted","Data":"66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc"}
Sep 10 20:58:21.749123 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:21.748728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw" event={"ID":"2c228527-cca0-4b08-a949-a138f8027dc1","Type":"ContainerStarted","Data":"a98c85681aac5777cbb69e2c25b1f70578ff76804be4ce133dfe25b482ee4d71"}
Sep 10 20:58:21.749123 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:21.748878    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:21.769668 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:21.769621    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw" podStartSLOduration=2.47534278 podStartE2EDuration="2.769607514s" totalImagesPullingTime="294.264734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:21.768994934 +0000 UTC m=+1337.680032218" watchObservedRunningTime="2026-09-10 20:58:21.769607514 +0000 UTC m=+1337.680644798"
Sep 10 20:58:23.916822 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:23.916787    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:58:27.759746 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:27.759657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 20:58:29.474972 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.474941    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"]
Sep 10 20:58:29.478870 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.478850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.483575 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.483544    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-3bbb40-serving-cert\""
Sep 10 20:58:29.495270 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.495250    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"]
Sep 10 20:58:29.556198 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.556173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.556338 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.556212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.556338 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.556238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k458g\" (UniqueName: \"kubernetes.io/projected/f3cab697-3320-40b7-88a4-ed45a4016c86-kube-api-access-k458g\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.556417 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.556377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cab697-3320-40b7-88a4-ed45a4016c86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.657127 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.657093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cab697-3320-40b7-88a4-ed45a4016c86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.657285 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.657140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.657285 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.657181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.657285 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.657214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k458g\" (UniqueName: \"kubernetes.io/projected/f3cab697-3320-40b7-88a4-ed45a4016c86-kube-api-access-k458g\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.657882 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.657851    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cab697-3320-40b7-88a4-ed45a4016c86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.659558 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.659532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.659816 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.659793    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.670365 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.670343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k458g\" (UniqueName: \"kubernetes.io/projected/f3cab697-3320-40b7-88a4-ed45a4016c86-kube-api-access-k458g\") pod \"maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:29.789668 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:29.789599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:30.080209 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:30.080183    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"]
Sep 10 20:58:30.082055 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:58:30.082025    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cab697_3320_40b7_88a4_ed45a4016c86.slice/crio-c4a77267a796787a7539bf95a3301f663b98aebfb4ea78d5b4fbf023cb3caf1c WatchSource:0}: Error finding container c4a77267a796787a7539bf95a3301f663b98aebfb4ea78d5b4fbf023cb3caf1c: Status 404 returned error can't find the container with id c4a77267a796787a7539bf95a3301f663b98aebfb4ea78d5b4fbf023cb3caf1c
Sep 10 20:58:30.806879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:30.806794    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2" event={"ID":"f3cab697-3320-40b7-88a4-ed45a4016c86","Type":"ContainerStarted","Data":"8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c"}
Sep 10 20:58:30.806879 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:30.806847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2" event={"ID":"f3cab697-3320-40b7-88a4-ed45a4016c86","Type":"ContainerStarted","Data":"c4a77267a796787a7539bf95a3301f663b98aebfb4ea78d5b4fbf023cb3caf1c"}
Sep 10 20:58:30.807293 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:30.806980    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:34.717221 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:34.717167    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2" podStartSLOduration=5.286228879 podStartE2EDuration="5.717149929s" totalImagesPullingTime="430.92105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:30.84510637 +0000 UTC m=+1346.756143653" watchObservedRunningTime="2026-09-10 20:58:34.717149929 +0000 UTC m=+1350.628187257"
Sep 10 20:58:34.717620 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:34.717491    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:58:36.818702 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:36.818657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 20:58:48.158758 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.158724    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"]
Sep 10 20:58:48.162840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.162818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.166121 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.166100    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f4fe4f-serving-cert\""
Sep 10 20:58:48.172783 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.172749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"]
Sep 10 20:58:48.221632 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.221600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.221770 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.221653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6d7\" (UniqueName: \"kubernetes.io/projected/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-kube-api-access-zg6d7\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.221815 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.221778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.221855 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.221816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.323194 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.323161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.323333 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.323207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.323333 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.323231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.323333 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.323276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6d7\" (UniqueName: \"kubernetes.io/projected/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-kube-api-access-zg6d7\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.323333 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:48.323305    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-serving-cert: secret "maas-api-e2e-shared-b-w3-f4fe4f-serving-cert" not found
Sep 10 20:58:48.323498 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:48.323379    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls podName:a85ae8e6-bba2-4deb-83ed-3276ff0d09a7 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:48.823363001 +0000 UTC m=+1364.734400263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" (UID: "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7") : secret "maas-api-e2e-shared-b-w3-f4fe4f-serving-cert" not found
Sep 10 20:58:48.323968 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.323949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.325709 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.325673    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.334124 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.334103    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6d7\" (UniqueName: \"kubernetes.io/projected/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-kube-api-access-zg6d7\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.828168 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:48.828124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:48.828352 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:48.828296    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-serving-cert: secret "maas-api-e2e-shared-b-w3-f4fe4f-serving-cert" not found
Sep 10 20:58:48.828396 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:48.828373    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls podName:a85ae8e6-bba2-4deb-83ed-3276ff0d09a7 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:49.828357924 +0000 UTC m=+1365.739395186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" (UID: "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7") : secret "maas-api-e2e-shared-b-w3-f4fe4f-serving-cert" not found
Sep 10 20:58:49.836899 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:49.836861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:49.839448 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:49.839412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:49.975853 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:49.975825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:50.250520 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:50.250489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"]
Sep 10 20:58:50.252447 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:58:50.252418    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85ae8e6_bba2_4deb_83ed_3276ff0d09a7.slice/crio-5129a7e0eb5cd1278cf3b44a19b05ac95ac15f88d120760c156ded48bcdda3bc WatchSource:0}: Error finding container 5129a7e0eb5cd1278cf3b44a19b05ac95ac15f88d120760c156ded48bcdda3bc: Status 404 returned error can't find the container with id 5129a7e0eb5cd1278cf3b44a19b05ac95ac15f88d120760c156ded48bcdda3bc
Sep 10 20:58:50.922259 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:50.922224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" event={"ID":"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7","Type":"ContainerStarted","Data":"c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed"}
Sep 10 20:58:50.922704 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:50.922279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" event={"ID":"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7","Type":"ContainerStarted","Data":"5129a7e0eb5cd1278cf3b44a19b05ac95ac15f88d120760c156ded48bcdda3bc"}
Sep 10 20:58:50.922704 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:50.922427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:50.945257 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:50.945211    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" podStartSLOduration=2.500949188 podStartE2EDuration="2.945196052s" totalImagesPullingTime="444.246864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:50.943601868 +0000 UTC m=+1366.854639151" watchObservedRunningTime="2026-09-10 20:58:50.945196052 +0000 UTC m=+1366.856233371"
Sep 10 20:58:53.639338 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.639308    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8017-9fc204"]
Sep 10 20:58:53.643290 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.643271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:58:53.646598 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.646574    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-64jfg\""
Sep 10 20:58:53.650240 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.650213    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-9fc204"]
Sep 10 20:58:53.773005 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.772975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwrg\" (UniqueName: \"kubernetes.io/projected/f53609cc-2cc7-45b3-891c-fc2084480697-kube-api-access-wdwrg\") pod \"test-curl-8017-9fc204\" (UID: \"f53609cc-2cc7-45b3-891c-fc2084480697\") " pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:58:53.873774 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.873744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwrg\" (UniqueName: \"kubernetes.io/projected/f53609cc-2cc7-45b3-891c-fc2084480697-kube-api-access-wdwrg\") pod \"test-curl-8017-9fc204\" (UID: \"f53609cc-2cc7-45b3-891c-fc2084480697\") " pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:58:53.883624 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.883604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwrg\" (UniqueName: \"kubernetes.io/projected/f53609cc-2cc7-45b3-891c-fc2084480697-kube-api-access-wdwrg\") pod \"test-curl-8017-9fc204\" (UID: \"f53609cc-2cc7-45b3-891c-fc2084480697\") " pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:58:53.961865 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:53.961807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:58:54.232640 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:54.232615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-9fc204"]
Sep 10 20:58:54.233529 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:58:54.233508    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf53609cc_2cc7_45b3_891c_fc2084480697.slice/crio-03337fcb14fce1ace40ec681aabfa4052244e73392645efe260364eda0526997 WatchSource:0}: Error finding container 03337fcb14fce1ace40ec681aabfa4052244e73392645efe260364eda0526997: Status 404 returned error can't find the container with id 03337fcb14fce1ace40ec681aabfa4052244e73392645efe260364eda0526997
Sep 10 20:58:54.819495 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:54.819461    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:58:54.946142 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:54.946104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-9fc204" event={"ID":"f53609cc-2cc7-45b3-891c-fc2084480697","Type":"ContainerStarted","Data":"d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538"}
Sep 10 20:58:54.946329 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:54.946153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-9fc204" event={"ID":"f53609cc-2cc7-45b3-891c-fc2084480697","Type":"ContainerStarted","Data":"03337fcb14fce1ace40ec681aabfa4052244e73392645efe260364eda0526997"}
Sep 10 20:58:54.984311 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:54.984260    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8017-9fc204" podStartSLOduration=1.9842451589999999 podStartE2EDuration="1.984245159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:54.982453616 +0000 UTC m=+1370.893490901" watchObservedRunningTime="2026-09-10 20:58:54.984245159 +0000 UTC m=+1370.895282443"
Sep 10 20:58:55.686243 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.686162    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8017-9fc204"]
Sep 10 20:58:55.697530 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.697498    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8017-9fc204"]
Sep 10 20:58:55.856524 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.856486    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8017-e78755"]
Sep 10 20:58:55.857154 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.857132    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53609cc-2cc7-45b3-891c-fc2084480697" containerName="test-curl-8017-9fc204"
Sep 10 20:58:55.857256 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.857156    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53609cc-2cc7-45b3-891c-fc2084480697" containerName="test-curl-8017-9fc204"
Sep 10 20:58:55.857318 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.857300    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53609cc-2cc7-45b3-891c-fc2084480697" containerName="test-curl-8017-9fc204"
Sep 10 20:58:55.860626 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.860603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:58:55.866223 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.866202    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-e78755"]
Sep 10 20:58:55.952522 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:55.952453    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8017-9fc204" container="test-curl-8017-9fc204"
Sep 10 20:58:55.994269 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:55.994225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpnk\" (UniqueName: \"kubernetes.io/projected/3f740963-19df-49bc-8432-17c1f592fffd-kube-api-access-ltpnk\") pod \"test-curl-8017-e78755\" (UID: \"3f740963-19df-49bc-8432-17c1f592fffd\") " pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:58:56.094968 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.094941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpnk\" (UniqueName: \"kubernetes.io/projected/3f740963-19df-49bc-8432-17c1f592fffd-kube-api-access-ltpnk\") pod \"test-curl-8017-e78755\" (UID: \"3f740963-19df-49bc-8432-17c1f592fffd\") " pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:58:56.106022 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.105994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpnk\" (UniqueName: \"kubernetes.io/projected/3f740963-19df-49bc-8432-17c1f592fffd-kube-api-access-ltpnk\") pod \"test-curl-8017-e78755\" (UID: \"3f740963-19df-49bc-8432-17c1f592fffd\") " pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:58:56.169920 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.169896    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:58:56.444420 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.444398    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-e78755"]
Sep 10 20:58:56.446705 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:58:56.446658    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f740963_19df_49bc_8432_17c1f592fffd.slice/crio-62e1f415e3f395f05abf1ee6455883e13481d23e920ab3d0a225973009e70cb9 WatchSource:0}: Error finding container 62e1f415e3f395f05abf1ee6455883e13481d23e920ab3d0a225973009e70cb9: Status 404 returned error can't find the container with id 62e1f415e3f395f05abf1ee6455883e13481d23e920ab3d0a225973009e70cb9
Sep 10 20:58:56.933571 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.933541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 20:58:56.960058 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.960026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-e78755" event={"ID":"3f740963-19df-49bc-8432-17c1f592fffd","Type":"ContainerStarted","Data":"a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c"}
Sep 10 20:58:56.960221 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.960068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-e78755" event={"ID":"3f740963-19df-49bc-8432-17c1f592fffd","Type":"ContainerStarted","Data":"62e1f415e3f395f05abf1ee6455883e13481d23e920ab3d0a225973009e70cb9"}
Sep 10 20:58:56.960221 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.960095    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8017-9fc204" podUID="f53609cc-2cc7-45b3-891c-fc2084480697" containerName="test-curl-8017-9fc204" containerID="cri-o://d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538" gracePeriod=2
Sep 10 20:58:56.980552 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:56.980508    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8017-e78755" podStartSLOduration=1.980493478 podStartE2EDuration="1.980493478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:56.979116899 +0000 UTC m=+1372.890154195" watchObservedRunningTime="2026-09-10 20:58:56.980493478 +0000 UTC m=+1372.891530762"
Sep 10 20:58:57.635165 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.635137    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8017-e78755"]
Sep 10 20:58:57.641267 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.641244    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8017-e78755"]
Sep 10 20:58:57.759784 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.757569    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8017-abdaec"]
Sep 10 20:58:57.759784 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.758714    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f740963-19df-49bc-8432-17c1f592fffd" containerName="test-curl-8017-e78755"
Sep 10 20:58:57.759784 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.758736    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f740963-19df-49bc-8432-17c1f592fffd" containerName="test-curl-8017-e78755"
Sep 10 20:58:57.759784 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.758952    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f740963-19df-49bc-8432-17c1f592fffd" containerName="test-curl-8017-e78755"
Sep 10 20:58:57.763402 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.763372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:58:57.768220 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.768197    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-abdaec"]
Sep 10 20:58:57.913972 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:57.913938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvklw\" (UniqueName: \"kubernetes.io/projected/6cebd5a4-4279-47e0-9ca1-fd58a25419d5-kube-api-access-cvklw\") pod \"test-curl-8017-abdaec\" (UID: \"6cebd5a4-4279-47e0-9ca1-fd58a25419d5\") " pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:58:57.966512 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:57.966487    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8017-e78755" container="test-curl-8017-e78755"
Sep 10 20:58:58.014543 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.014483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvklw\" (UniqueName: \"kubernetes.io/projected/6cebd5a4-4279-47e0-9ca1-fd58a25419d5-kube-api-access-cvklw\") pod \"test-curl-8017-abdaec\" (UID: \"6cebd5a4-4279-47e0-9ca1-fd58a25419d5\") " pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:58:58.025809 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.025778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvklw\" (UniqueName: \"kubernetes.io/projected/6cebd5a4-4279-47e0-9ca1-fd58a25419d5-kube-api-access-cvklw\") pod \"test-curl-8017-abdaec\" (UID: \"6cebd5a4-4279-47e0-9ca1-fd58a25419d5\") " pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:58:58.074371 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.074336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:58:58.346331 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.346305    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-abdaec"]
Sep 10 20:58:58.348167 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:58:58.348138    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cebd5a4_4279_47e0_9ca1_fd58a25419d5.slice/crio-025c61e136fd2813593ce2236a9607a4a1922a4fc22a5b1bb4995d533424c9ae WatchSource:0}: Error finding container 025c61e136fd2813593ce2236a9607a4a1922a4fc22a5b1bb4995d533424c9ae: Status 404 returned error can't find the container with id 025c61e136fd2813593ce2236a9607a4a1922a4fc22a5b1bb4995d533424c9ae
Sep 10 20:58:58.973464 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.973437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-abdaec" event={"ID":"6cebd5a4-4279-47e0-9ca1-fd58a25419d5","Type":"ContainerStarted","Data":"80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7"}
Sep 10 20:58:58.973782 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.973483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-abdaec" event={"ID":"6cebd5a4-4279-47e0-9ca1-fd58a25419d5","Type":"ContainerStarted","Data":"025c61e136fd2813593ce2236a9607a4a1922a4fc22a5b1bb4995d533424c9ae"}
Sep 10 20:58:58.973782 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.973608    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8017-e78755" podUID="3f740963-19df-49bc-8432-17c1f592fffd" containerName="test-curl-8017-e78755" containerID="cri-o://a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c" gracePeriod=2
Sep 10 20:58:58.995595 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:58.995550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8017-abdaec" podStartSLOduration=1.995534672 podStartE2EDuration="1.995534672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:58.992718988 +0000 UTC m=+1374.903756271" watchObservedRunningTime="2026-09-10 20:58:58.995534672 +0000 UTC m=+1374.906571957"
Sep 10 20:58:59.131444 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.131421    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:58:59.226408 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.226353    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdwrg\" (UniqueName: \"kubernetes.io/projected/f53609cc-2cc7-45b3-891c-fc2084480697-kube-api-access-wdwrg\") pod \"f53609cc-2cc7-45b3-891c-fc2084480697\" (UID: \"f53609cc-2cc7-45b3-891c-fc2084480697\") "
Sep 10 20:58:59.228147 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.228124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53609cc-2cc7-45b3-891c-fc2084480697-kube-api-access-wdwrg" (OuterVolumeSpecName: "kube-api-access-wdwrg") pod "f53609cc-2cc7-45b3-891c-fc2084480697" (UID: "f53609cc-2cc7-45b3-891c-fc2084480697"). InnerVolumeSpecName "kube-api-access-wdwrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:58:59.327381 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.327354    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwrg\" (UniqueName: \"kubernetes.io/projected/f53609cc-2cc7-45b3-891c-fc2084480697-kube-api-access-wdwrg\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:58:59.590049 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.590017    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8017-abdaec"]
Sep 10 20:58:59.592337 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.592309    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8017-abdaec"]
Sep 10 20:58:59.716788 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.716758    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8017-34d0b5"]
Sep 10 20:58:59.717237 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.717223    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" containerName="test-curl-8017-abdaec"
Sep 10 20:58:59.717284 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.717238    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" containerName="test-curl-8017-abdaec"
Sep 10 20:58:59.717323 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.717314    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" containerName="test-curl-8017-abdaec"
Sep 10 20:58:59.720316 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.720299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:58:59.728757 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.728732    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-34d0b5"]
Sep 10 20:58:59.831420 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.831393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wpq\" (UniqueName: \"kubernetes.io/projected/efc865e3-60fe-4671-8a70-f50f48dadebe-kube-api-access-s6wpq\") pod \"test-curl-8017-34d0b5\" (UID: \"efc865e3-60fe-4671-8a70-f50f48dadebe\") " pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:58:59.932813 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.932742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wpq\" (UniqueName: \"kubernetes.io/projected/efc865e3-60fe-4671-8a70-f50f48dadebe-kube-api-access-s6wpq\") pod \"test-curl-8017-34d0b5\" (UID: \"efc865e3-60fe-4671-8a70-f50f48dadebe\") " pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:58:59.949100 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.949069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wpq\" (UniqueName: \"kubernetes.io/projected/efc865e3-60fe-4671-8a70-f50f48dadebe-kube-api-access-s6wpq\") pod \"test-curl-8017-34d0b5\" (UID: \"efc865e3-60fe-4671-8a70-f50f48dadebe\") " pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:58:59.980941 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.980911    2565 generic.go:356] "Generic (PLEG): container finished" podID="f53609cc-2cc7-45b3-891c-fc2084480697" containerID="d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538" exitCode=137
Sep 10 20:58:59.981305 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.980961    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:58:59.981305 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.980999    2565 scope.go:117] "RemoveContainer" containerID="d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538"
Sep 10 20:58:59.981433 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:59.981319    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8017-abdaec" container="test-curl-8017-abdaec"
Sep 10 20:58:59.991068 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.991051    2565 scope.go:117] "RemoveContainer" containerID="d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538"
Sep 10 20:58:59.991324 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:59.991307    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538\": container with ID starting with d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538 not found: ID does not exist" containerID="d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538"
Sep 10 20:58:59.991378 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:58:59.991332    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538"} err="failed to get container status \"d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538\": rpc error: code = NotFound desc = could not find container \"d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538\": container with ID starting with d4e4d2a5c5befcb71886b41c25a11d7742251d4c027755e37a9fd2979c51a538 not found: ID does not exist"
Sep 10 20:58:59.993837 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:58:59.993814    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8017-9fc204\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="f53609cc-2cc7-45b3-891c-fc2084480697" pod="opendatahub/test-curl-8017-9fc204"
Sep 10 20:59:00.031019 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:00.030997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:59:00.308572 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:00.308543    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8017-34d0b5"]
Sep 10 20:59:00.309445 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:59:00.309419    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefc865e3_60fe_4671_8a70_f50f48dadebe.slice/crio-95768ffe1bcbca41358d44339dc3abae8abee8a32bb5345e80a5ab5b1631ec28 WatchSource:0}: Error finding container 95768ffe1bcbca41358d44339dc3abae8abee8a32bb5345e80a5ab5b1631ec28: Status 404 returned error can't find the container with id 95768ffe1bcbca41358d44339dc3abae8abee8a32bb5345e80a5ab5b1631ec28
Sep 10 20:59:00.591507 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:00.591475    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f53609cc-2cc7-45b3-891c-fc2084480697" path="/var/lib/kubelet/pods/f53609cc-2cc7-45b3-891c-fc2084480697/volumes"
Sep 10 20:59:00.989745 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:00.989716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-34d0b5" event={"ID":"efc865e3-60fe-4671-8a70-f50f48dadebe","Type":"ContainerStarted","Data":"04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1"}
Sep 10 20:59:00.990109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:00.989768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8017-34d0b5" event={"ID":"efc865e3-60fe-4671-8a70-f50f48dadebe","Type":"ContainerStarted","Data":"95768ffe1bcbca41358d44339dc3abae8abee8a32bb5345e80a5ab5b1631ec28"}
Sep 10 20:59:00.991031 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:00.990989    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8017-abdaec" podUID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" containerName="test-curl-8017-abdaec" containerID="cri-o://80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7" gracePeriod=2
Sep 10 20:59:01.010231 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.010182    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8017-34d0b5" podStartSLOduration=2.0101666 podStartE2EDuration="2.0101666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:01.009155225 +0000 UTC m=+1376.920192510" watchObservedRunningTime="2026-09-10 20:59:01.0101666 +0000 UTC m=+1376.921203883"
Sep 10 20:59:01.124567 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.124543    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:59:01.245494 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.245426    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltpnk\" (UniqueName: \"kubernetes.io/projected/3f740963-19df-49bc-8432-17c1f592fffd-kube-api-access-ltpnk\") pod \"3f740963-19df-49bc-8432-17c1f592fffd\" (UID: \"3f740963-19df-49bc-8432-17c1f592fffd\") "
Sep 10 20:59:01.247280 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.247252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f740963-19df-49bc-8432-17c1f592fffd-kube-api-access-ltpnk" (OuterVolumeSpecName: "kube-api-access-ltpnk") pod "3f740963-19df-49bc-8432-17c1f592fffd" (UID: "3f740963-19df-49bc-8432-17c1f592fffd"). InnerVolumeSpecName "kube-api-access-ltpnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:01.346945 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.346919    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltpnk\" (UniqueName: \"kubernetes.io/projected/3f740963-19df-49bc-8432-17c1f592fffd-kube-api-access-ltpnk\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:59:01.627814 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.627785    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8017-34d0b5"]
Sep 10 20:59:01.633578 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.633550    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8017-34d0b5"]
Sep 10 20:59:01.998137 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.998046    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f740963-19df-49bc-8432-17c1f592fffd" containerID="a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c" exitCode=137
Sep 10 20:59:01.998137 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.998087    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:59:01.998617 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:01.998139    2565 scope.go:117] "RemoveContainer" containerID="a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c"
Sep 10 20:59:01.998617 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:01.998443    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8017-34d0b5" container="test-curl-8017-34d0b5"
Sep 10 20:59:02.007678 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:02.007662    2565 scope.go:117] "RemoveContainer" containerID="a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c"
Sep 10 20:59:02.007979 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:02.007959    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c\": container with ID starting with a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c not found: ID does not exist" containerID="a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c"
Sep 10 20:59:02.008042 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:02.007985    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c"} err="failed to get container status \"a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c\": rpc error: code = NotFound desc = could not find container \"a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c\": container with ID starting with a0f80f8daad40ff193f360d8e29365bb3762d1a361b47c12cc855ed1dd6c6b6c not found: ID does not exist"
Sep 10 20:59:02.010034 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:02.010013    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8017-e78755\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="3f740963-19df-49bc-8432-17c1f592fffd" pod="opendatahub/test-curl-8017-e78755"
Sep 10 20:59:02.591276 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:02.591244    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f740963-19df-49bc-8432-17c1f592fffd" path="/var/lib/kubelet/pods/3f740963-19df-49bc-8432-17c1f592fffd/volumes"
Sep 10 20:59:03.008696 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:03.008655    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8017-34d0b5" podUID="efc865e3-60fe-4671-8a70-f50f48dadebe" containerName="test-curl-8017-34d0b5" containerID="cri-o://04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1" gracePeriod=2
Sep 10 20:59:03.171665 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:03.171642    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:59:03.262926 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:03.262865    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvklw\" (UniqueName: \"kubernetes.io/projected/6cebd5a4-4279-47e0-9ca1-fd58a25419d5-kube-api-access-cvklw\") pod \"6cebd5a4-4279-47e0-9ca1-fd58a25419d5\" (UID: \"6cebd5a4-4279-47e0-9ca1-fd58a25419d5\") "
Sep 10 20:59:03.264840 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:03.264818    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cebd5a4-4279-47e0-9ca1-fd58a25419d5-kube-api-access-cvklw" (OuterVolumeSpecName: "kube-api-access-cvklw") pod "6cebd5a4-4279-47e0-9ca1-fd58a25419d5" (UID: "6cebd5a4-4279-47e0-9ca1-fd58a25419d5"). InnerVolumeSpecName "kube-api-access-cvklw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:03.364494 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:03.364467    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvklw\" (UniqueName: \"kubernetes.io/projected/6cebd5a4-4279-47e0-9ca1-fd58a25419d5-kube-api-access-cvklw\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:59:04.015621 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:04.015581    2565 generic.go:356] "Generic (PLEG): container finished" podID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" containerID="80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7" exitCode=137
Sep 10 20:59:04.015621 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:04.015622    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:59:04.018252 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:04.015681    2565 scope.go:117] "RemoveContainer" containerID="80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7"
Sep 10 20:59:04.025327 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:04.025308    2565 scope.go:117] "RemoveContainer" containerID="80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7"
Sep 10 20:59:04.025774 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:04.025740    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7\": container with ID starting with 80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7 not found: ID does not exist" containerID="80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7"
Sep 10 20:59:04.025909 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:04.025881    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7"} err="failed to get container status \"80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7\": rpc error: code = NotFound desc = could not find container \"80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7\": container with ID starting with 80a4f9d10323e8c6806da0b64454851c7ee080ca2cd28253ddbbcd9e06115ca7 not found: ID does not exist"
Sep 10 20:59:04.027878 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:04.027856    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8017-abdaec\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:59:04.591357 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:04.591326    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" path="/var/lib/kubelet/pods/6cebd5a4-4279-47e0-9ca1-fd58a25419d5/volumes"
Sep 10 20:59:04.592359 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:04.592336    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8017-abdaec\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="6cebd5a4-4279-47e0-9ca1-fd58a25419d5" pod="opendatahub/test-curl-8017-abdaec"
Sep 10 20:59:04.762758 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:04.762728    2565 scope.go:117] "RemoveContainer" containerID="e97ed30e4a81b829bf72603a120d3f54bacbb134b4f5a61b3c6bb35c39ecc337"
Sep 10 20:59:05.190027 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:05.190002    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:59:05.281109 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:05.281043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6wpq\" (UniqueName: \"kubernetes.io/projected/efc865e3-60fe-4671-8a70-f50f48dadebe-kube-api-access-s6wpq\") pod \"efc865e3-60fe-4671-8a70-f50f48dadebe\" (UID: \"efc865e3-60fe-4671-8a70-f50f48dadebe\") "
Sep 10 20:59:05.283047 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:05.283023    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc865e3-60fe-4671-8a70-f50f48dadebe-kube-api-access-s6wpq" (OuterVolumeSpecName: "kube-api-access-s6wpq") pod "efc865e3-60fe-4671-8a70-f50f48dadebe" (UID: "efc865e3-60fe-4671-8a70-f50f48dadebe"). InnerVolumeSpecName "kube-api-access-s6wpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:05.381977 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:05.381948    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wpq\" (UniqueName: \"kubernetes.io/projected/efc865e3-60fe-4671-8a70-f50f48dadebe-kube-api-access-s6wpq\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 20:59:06.028730 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:06.028697    2565 generic.go:356] "Generic (PLEG): container finished" podID="efc865e3-60fe-4671-8a70-f50f48dadebe" containerID="04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1" exitCode=137
Sep 10 20:59:06.028916 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:06.028745    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:59:06.028916 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:06.028769    2565 scope.go:117] "RemoveContainer" containerID="04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1"
Sep 10 20:59:06.038909 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:06.038892    2565 scope.go:117] "RemoveContainer" containerID="04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1"
Sep 10 20:59:06.039197 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:06.039180    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1\": container with ID starting with 04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1 not found: ID does not exist" containerID="04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1"
Sep 10 20:59:06.039257 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:06.039202    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1"} err="failed to get container status \"04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1\": rpc error: code = NotFound desc = could not find container \"04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1\": container with ID starting with 04509d9238d9f1cf43e55a4efc8df97205fcc6c559ab01aadad35bb186ede4d1 not found: ID does not exist"
Sep 10 20:59:06.055886 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:06.055855    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8017-34d0b5\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="efc865e3-60fe-4671-8a70-f50f48dadebe" pod="opendatahub/test-curl-8017-34d0b5"
Sep 10 20:59:06.591995 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:06.591961    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc865e3-60fe-4671-8a70-f50f48dadebe" path="/var/lib/kubelet/pods/efc865e3-60fe-4671-8a70-f50f48dadebe/volumes"
Sep 10 20:59:07.017797 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.017712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"]
Sep 10 20:59:07.018358 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.018337    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc865e3-60fe-4671-8a70-f50f48dadebe" containerName="test-curl-8017-34d0b5"
Sep 10 20:59:07.018358 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.018360    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc865e3-60fe-4671-8a70-f50f48dadebe" containerName="test-curl-8017-34d0b5"
Sep 10 20:59:07.018492 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.018438    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc865e3-60fe-4671-8a70-f50f48dadebe" containerName="test-curl-8017-34d0b5"
Sep 10 20:59:07.022987 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.022964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.032681 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.032659    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d4d0c56-serving-cert\""
Sep 10 20:59:07.048258 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.048232    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"]
Sep 10 20:59:07.199256 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.199223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.199420 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.199281    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kxjl\" (UniqueName: \"kubernetes.io/projected/29671b2b-9031-46c4-b820-39b035c8b543-kube-api-access-9kxjl\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.199420 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.199349    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29671b2b-9031-46c4-b820-39b035c8b543-openshift-service-ca\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.199420 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.199380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-metrics-certs\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.300079 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.299994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.300079 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.300059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kxjl\" (UniqueName: \"kubernetes.io/projected/29671b2b-9031-46c4-b820-39b035c8b543-kube-api-access-9kxjl\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.300262 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.300114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29671b2b-9031-46c4-b820-39b035c8b543-openshift-service-ca\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.300262 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.300129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-metrics-certs\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.300262 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:07.300147    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-serving-cert: secret "maas-api-e2e-mt-3d4d0c56-serving-cert" not found
Sep 10 20:59:07.300262 ip-10-0-129-103 kubenswrapper[2565]: E0910 20:59:07.300211    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls podName:29671b2b-9031-46c4-b820-39b035c8b543 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:07.800194592 +0000 UTC m=+1383.711231855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls") pod "maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68" (UID: "29671b2b-9031-46c4-b820-39b035c8b543") : secret "maas-api-e2e-mt-3d4d0c56-serving-cert" not found
Sep 10 20:59:07.300858 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.300839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29671b2b-9031-46c4-b820-39b035c8b543-openshift-service-ca\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.302549 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.302532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-metrics-certs\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.323354 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.323324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kxjl\" (UniqueName: \"kubernetes.io/projected/29671b2b-9031-46c4-b820-39b035c8b543-kube-api-access-9kxjl\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.804499 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.804466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.806812 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.806789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls\") pod \"maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:07.934455 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:07.934424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:08.222760 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:08.222728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"]
Sep 10 20:59:08.222939 ip-10-0-129-103 kubenswrapper[2565]: W0910 20:59:08.222914    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29671b2b_9031_46c4_b820_39b035c8b543.slice/crio-a3c87a15b95b4c6392d811d50ba4f94b3abc42e8e0a2be99016c7357afdda471 WatchSource:0}: Error finding container a3c87a15b95b4c6392d811d50ba4f94b3abc42e8e0a2be99016c7357afdda471: Status 404 returned error can't find the container with id a3c87a15b95b4c6392d811d50ba4f94b3abc42e8e0a2be99016c7357afdda471
Sep 10 20:59:09.052045 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:09.051946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68" event={"ID":"29671b2b-9031-46c4-b820-39b035c8b543","Type":"ContainerStarted","Data":"1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c"}
Sep 10 20:59:09.052045 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:09.052001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68" event={"ID":"29671b2b-9031-46c4-b820-39b035c8b543","Type":"ContainerStarted","Data":"a3c87a15b95b4c6392d811d50ba4f94b3abc42e8e0a2be99016c7357afdda471"}
Sep 10 20:59:09.052045 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:09.052023    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:13.137795 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:13.137737    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68" podStartSLOduration=6.754523462 podStartE2EDuration="7.137717107s" totalImagesPullingTime="383.193645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:09.117980201 +0000 UTC m=+1385.029017485" watchObservedRunningTime="2026-09-10 20:59:13.137717107 +0000 UTC m=+1389.048754392"
Sep 10 20:59:13.139180 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:13.139158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 20:59:15.063468 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:15.063440    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 20:59:27.630467 ip-10-0-129-103 kubenswrapper[2565]: I0910 20:59:27.630430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:00:00.801471 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:00:00.801397    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:00:04.798250 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:00:04.798214    2565 scope.go:117] "RemoveContainer" containerID="f27a6d4976217fe7cea8ced6a1fb18d84c412a05c667ab7179fadeac1e417609"
Sep 10 21:00:22.199845 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:00:22.199810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:00:39.524989 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:00:39.524948    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:01:01.019209 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:01.019177    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:01:01.974854 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:01.974815    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"]
Sep 10 21:01:01.975058 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:01.975030    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68" podUID="29671b2b-9031-46c4-b820-39b035c8b543" containerName="maas-api" containerID="cri-o://1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c" gracePeriod=30
Sep 10 21:01:02.234545 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.234480    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 21:01:02.329264 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.329234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29671b2b-9031-46c4-b820-39b035c8b543-openshift-service-ca\") pod \"29671b2b-9031-46c4-b820-39b035c8b543\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") "
Sep 10 21:01:02.329414 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.329401    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls\") pod \"29671b2b-9031-46c4-b820-39b035c8b543\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") "
Sep 10 21:01:02.329585 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.329445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kxjl\" (UniqueName: \"kubernetes.io/projected/29671b2b-9031-46c4-b820-39b035c8b543-kube-api-access-9kxjl\") pod \"29671b2b-9031-46c4-b820-39b035c8b543\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") "
Sep 10 21:01:02.329585 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.329475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-metrics-certs\") pod \"29671b2b-9031-46c4-b820-39b035c8b543\" (UID: \"29671b2b-9031-46c4-b820-39b035c8b543\") "
Sep 10 21:01:02.329724 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.329613    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29671b2b-9031-46c4-b820-39b035c8b543-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29671b2b-9031-46c4-b820-39b035c8b543" (UID: "29671b2b-9031-46c4-b820-39b035c8b543"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:02.329826 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.329806    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29671b2b-9031-46c4-b820-39b035c8b543-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:01:02.331598 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.331563    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29671b2b-9031-46c4-b820-39b035c8b543" (UID: "29671b2b-9031-46c4-b820-39b035c8b543"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:01:02.331730 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.331613    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29671b2b-9031-46c4-b820-39b035c8b543-kube-api-access-9kxjl" (OuterVolumeSpecName: "kube-api-access-9kxjl") pod "29671b2b-9031-46c4-b820-39b035c8b543" (UID: "29671b2b-9031-46c4-b820-39b035c8b543"). InnerVolumeSpecName "kube-api-access-9kxjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:01:02.331730 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.331625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29671b2b-9031-46c4-b820-39b035c8b543" (UID: "29671b2b-9031-46c4-b820-39b035c8b543"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:01:02.430506 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.430479    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:01:02.430506 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.430503    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kxjl\" (UniqueName: \"kubernetes.io/projected/29671b2b-9031-46c4-b820-39b035c8b543-kube-api-access-9kxjl\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:01:02.430673 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.430513    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29671b2b-9031-46c4-b820-39b035c8b543-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:01:02.697147 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.697115    2565 generic.go:356] "Generic (PLEG): container finished" podID="29671b2b-9031-46c4-b820-39b035c8b543" containerID="1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c" exitCode=0
Sep 10 21:01:02.697293 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.697164    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"
Sep 10 21:01:02.697293 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.697210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68" event={"ID":"29671b2b-9031-46c4-b820-39b035c8b543","Type":"ContainerDied","Data":"1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c"}
Sep 10 21:01:02.697293 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.697269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68" event={"ID":"29671b2b-9031-46c4-b820-39b035c8b543","Type":"ContainerDied","Data":"a3c87a15b95b4c6392d811d50ba4f94b3abc42e8e0a2be99016c7357afdda471"}
Sep 10 21:01:02.697442 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.697302    2565 scope.go:117] "RemoveContainer" containerID="1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c"
Sep 10 21:01:02.710138 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.710122    2565 scope.go:117] "RemoveContainer" containerID="1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c"
Sep 10 21:01:02.710394 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:01:02.710378    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c\": container with ID starting with 1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c not found: ID does not exist" containerID="1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c"
Sep 10 21:01:02.710434 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.710403    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c"} err="failed to get container status \"1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c\": rpc error: code = NotFound desc = could not find container \"1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c\": container with ID starting with 1a01e7b060829182ae0191a31826cbfda1fa4e5f5253ee8bd325b738ac78235c not found: ID does not exist"
Sep 10 21:01:02.720461 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.720434    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"]
Sep 10 21:01:02.723104 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:02.723084    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d4d0c56-85dc458f5-48p68"]
Sep 10 21:01:04.592777 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.592748    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29671b2b-9031-46c4-b820-39b035c8b543" path="/var/lib/kubelet/pods/29671b2b-9031-46c4-b820-39b035c8b543/volumes"
Sep 10 21:01:04.658435 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.658409    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:01:04.659583 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.659557    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:01:04.662192 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.662175    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:01:04.663314 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.663291    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:01:04.824331 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.824307    2565 scope.go:117] "RemoveContainer" containerID="9436b1b4e1fe9982faa68ba49625e347986d807bec254065572aa9ffe59edf7d"
Sep 10 21:01:04.832776 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.832760    2565 scope.go:117] "RemoveContainer" containerID="2106647d2efff475b7cd99bdc69badd9008b3af0a6e14559c6ce93f5b964b534"
Sep 10 21:01:04.841650 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:04.841634    2565 scope.go:117] "RemoveContainer" containerID="a1251715bca59fba69f1842569a732916c4ab96874afbd285dbd074ee872a8b0"
Sep 10 21:01:14.504372 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:14.504338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:01:30.296540 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:30.296463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:01:46.614029 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:01:46.613997    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:02:02.419930 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:02.419894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:02:15.719317 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:15.719276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:02:30.800554 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:30.800516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:02:40.045923 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.045884    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8020-5de578"]
Sep 10 21:02:40.046449 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.046429    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29671b2b-9031-46c4-b820-39b035c8b543" containerName="maas-api"
Sep 10 21:02:40.046449 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.046449    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29671b2b-9031-46c4-b820-39b035c8b543" containerName="maas-api"
Sep 10 21:02:40.046554 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.046544    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29671b2b-9031-46c4-b820-39b035c8b543" containerName="maas-api"
Sep 10 21:02:40.049718 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.049676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:40.052740 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.052720    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-64jfg\""
Sep 10 21:02:40.057853 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.057270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8020-5de578"]
Sep 10 21:02:40.101093 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.101059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqvn\" (UniqueName: \"kubernetes.io/projected/51f11aea-009a-4193-bd8f-bc4b8cac3bab-kube-api-access-5mqvn\") pod \"test-curl-8020-5de578\" (UID: \"51f11aea-009a-4193-bd8f-bc4b8cac3bab\") " pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:40.202216 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.202187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqvn\" (UniqueName: \"kubernetes.io/projected/51f11aea-009a-4193-bd8f-bc4b8cac3bab-kube-api-access-5mqvn\") pod \"test-curl-8020-5de578\" (UID: \"51f11aea-009a-4193-bd8f-bc4b8cac3bab\") " pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:40.211022 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.210998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqvn\" (UniqueName: \"kubernetes.io/projected/51f11aea-009a-4193-bd8f-bc4b8cac3bab-kube-api-access-5mqvn\") pod \"test-curl-8020-5de578\" (UID: \"51f11aea-009a-4193-bd8f-bc4b8cac3bab\") " pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:40.360774 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.360738    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:40.630304 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:40.630276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8020-5de578"]
Sep 10 21:02:40.631543 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:02:40.631511    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f11aea_009a_4193_bd8f_bc4b8cac3bab.slice/crio-0da87bad8a04e0953ff88afcb2e86f16059b86d55d3146fce28e5409032809a3 WatchSource:0}: Error finding container 0da87bad8a04e0953ff88afcb2e86f16059b86d55d3146fce28e5409032809a3: Status 404 returned error can't find the container with id 0da87bad8a04e0953ff88afcb2e86f16059b86d55d3146fce28e5409032809a3
Sep 10 21:02:41.249236 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:41.249204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8020-5de578" event={"ID":"51f11aea-009a-4193-bd8f-bc4b8cac3bab","Type":"ContainerStarted","Data":"3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7"}
Sep 10 21:02:41.249732 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:41.249245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8020-5de578" event={"ID":"51f11aea-009a-4193-bd8f-bc4b8cac3bab","Type":"ContainerStarted","Data":"0da87bad8a04e0953ff88afcb2e86f16059b86d55d3146fce28e5409032809a3"}
Sep 10 21:02:41.272544 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:41.272501    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8020-5de578" podStartSLOduration=1.272488082 podStartE2EDuration="1.272488082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:41.270062829 +0000 UTC m=+1597.181100112" watchObservedRunningTime="2026-09-10 21:02:41.272488082 +0000 UTC m=+1597.183525366"
Sep 10 21:02:41.944020 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:41.943991    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8020-5de578"]
Sep 10 21:02:41.951259 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:41.951206    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8020-5de578"]
Sep 10 21:02:42.255522 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:02:42.255492    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8020-5de578" container="test-curl-8020-5de578"
Sep 10 21:02:43.261254 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:43.261211    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8020-5de578" podUID="51f11aea-009a-4193-bd8f-bc4b8cac3bab" containerName="test-curl-8020-5de578" containerID="cri-o://3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7" gracePeriod=2
Sep 10 21:02:45.428415 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:45.428391    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:45.449958 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:45.449935    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mqvn\" (UniqueName: \"kubernetes.io/projected/51f11aea-009a-4193-bd8f-bc4b8cac3bab-kube-api-access-5mqvn\") pod \"51f11aea-009a-4193-bd8f-bc4b8cac3bab\" (UID: \"51f11aea-009a-4193-bd8f-bc4b8cac3bab\") "
Sep 10 21:02:45.451852 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:45.451830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f11aea-009a-4193-bd8f-bc4b8cac3bab-kube-api-access-5mqvn" (OuterVolumeSpecName: "kube-api-access-5mqvn") pod "51f11aea-009a-4193-bd8f-bc4b8cac3bab" (UID: "51f11aea-009a-4193-bd8f-bc4b8cac3bab"). InnerVolumeSpecName "kube-api-access-5mqvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:02:45.551155 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:45.551099    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mqvn\" (UniqueName: \"kubernetes.io/projected/51f11aea-009a-4193-bd8f-bc4b8cac3bab-kube-api-access-5mqvn\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.005654 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:46.005622    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:02:46.280158 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:46.280073    2565 generic.go:356] "Generic (PLEG): container finished" podID="51f11aea-009a-4193-bd8f-bc4b8cac3bab" containerID="3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7" exitCode=137
Sep 10 21:02:46.280158 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:46.280112    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:46.280158 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:46.280139    2565 scope.go:117] "RemoveContainer" containerID="3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7"
Sep 10 21:02:46.289750 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:46.289727    2565 scope.go:117] "RemoveContainer" containerID="3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7"
Sep 10 21:02:46.290014 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:02:46.289995    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7\": container with ID starting with 3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7 not found: ID does not exist" containerID="3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7"
Sep 10 21:02:46.290079 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:46.290022    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7"} err="failed to get container status \"3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7\": rpc error: code = NotFound desc = could not find container \"3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7\": container with ID starting with 3dc7e21daecb7e93b4cb9d47b24db124160047071e1c22dd718a0c50a45475c7 not found: ID does not exist"
Sep 10 21:02:46.291568 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:02:46.291546    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8020-5de578\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="51f11aea-009a-4193-bd8f-bc4b8cac3bab" pod="opendatahub/test-curl-8020-5de578"
Sep 10 21:02:46.591127 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:46.591099    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f11aea-009a-4193-bd8f-bc4b8cac3bab" path="/var/lib/kubelet/pods/51f11aea-009a-4193-bd8f-bc4b8cac3bab/volumes"
Sep 10 21:02:56.220354 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.220305    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"]
Sep 10 21:02:56.221058 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.221034    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f11aea-009a-4193-bd8f-bc4b8cac3bab" containerName="test-curl-8020-5de578"
Sep 10 21:02:56.221058 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.221058    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f11aea-009a-4193-bd8f-bc4b8cac3bab" containerName="test-curl-8020-5de578"
Sep 10 21:02:56.221229 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.221151    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f11aea-009a-4193-bd8f-bc4b8cac3bab" containerName="test-curl-8020-5de578"
Sep 10 21:02:56.225851 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.225828    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.239909 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.239886    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d66961-serving-cert\""
Sep 10 21:02:56.242545 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.242519    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"]
Sep 10 21:02:56.341531 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.341495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplt7\" (UniqueName: \"kubernetes.io/projected/b02e728a-ac8a-435b-b13c-55b836010998-kube-api-access-pplt7\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.341671 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.341545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-metrics-certs\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.341671 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.341634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.341791 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.341751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e728a-ac8a-435b-b13c-55b836010998-openshift-service-ca\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.442949 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.442920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-metrics-certs\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.443117 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.442968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.443117 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.443039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e728a-ac8a-435b-b13c-55b836010998-openshift-service-ca\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.443117 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.443084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplt7\" (UniqueName: \"kubernetes.io/projected/b02e728a-ac8a-435b-b13c-55b836010998-kube-api-access-pplt7\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.443313 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:02:56.443163    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-serving-cert: secret "maas-api-e2e-api-a-d66961-serving-cert" not found
Sep 10 21:02:56.443313 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:02:56.443252    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls podName:b02e728a-ac8a-435b-b13c-55b836010998 nodeName:}" failed. No retries permitted until 2026-09-10 21:02:56.943230438 +0000 UTC m=+1612.854267714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls") pod "maas-api-e2e-api-a-d66961-5cf5866f97-57lrd" (UID: "b02e728a-ac8a-435b-b13c-55b836010998") : secret "maas-api-e2e-api-a-d66961-serving-cert" not found
Sep 10 21:02:56.443717 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.443677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e728a-ac8a-435b-b13c-55b836010998-openshift-service-ca\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.445391 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.445375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-metrics-certs\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.464948 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.464920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplt7\" (UniqueName: \"kubernetes.io/projected/b02e728a-ac8a-435b-b13c-55b836010998-kube-api-access-pplt7\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.948471 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.948377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:56.950858 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:56.950827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls\") pod \"maas-api-e2e-api-a-d66961-5cf5866f97-57lrd\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:57.139183 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.139149    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:57.399338 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.399307    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"]
Sep 10 21:02:57.403273 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.403250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.409803 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.409782    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"default-dockercfg-4hgln\""
Sep 10 21:02:57.411508 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.411477    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"sub-test-model-3fc2b8-kserve-self-signed-certs\""
Sep 10 21:02:57.411508 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.411488    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"openshift-service-ca.crt\""
Sep 10 21:02:57.411662 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.411531    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"kube-root-ca.crt\""
Sep 10 21:02:57.421906 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.421887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"]
Sep 10 21:02:57.427106 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.427066    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"]
Sep 10 21:02:57.429625 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:02:57.429598    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02e728a_ac8a_435b_b13c_55b836010998.slice/crio-d58835e0caf74475001e164739c993b97b50f51f3e486f11da354b33f6275a9c WatchSource:0}: Error finding container d58835e0caf74475001e164739c993b97b50f51f3e486f11da354b33f6275a9c: Status 404 returned error can't find the container with id d58835e0caf74475001e164739c993b97b50f51f3e486f11da354b33f6275a9c
Sep 10 21:02:57.431469 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.431451    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:02:57.453857 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.453830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-tmp-dir\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.453980 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.453921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b545ed-1470-4694-a100-079996b76069-tls-certs\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.454050 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.454009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrlvs\" (UniqueName: \"kubernetes.io/projected/f8b545ed-1470-4694-a100-079996b76069-kube-api-access-vrlvs\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.454120 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.454101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-home\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.454179 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.454155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-dshm\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.454233 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.454204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-model-cache\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555382 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-dshm\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555524 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-model-cache\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555524 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-tmp-dir\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555524 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b545ed-1470-4694-a100-079996b76069-tls-certs\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555524 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlvs\" (UniqueName: \"kubernetes.io/projected/f8b545ed-1470-4694-a100-079996b76069-kube-api-access-vrlvs\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555779 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-home\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555907 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-tmp-dir\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555973 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-model-cache\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.555973 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.555942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-home\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.557679 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.557658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-dshm\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.557986 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.557957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b545ed-1470-4694-a100-079996b76069-tls-certs\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.567964 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.567944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlvs\" (UniqueName: \"kubernetes.io/projected/f8b545ed-1470-4694-a100-079996b76069-kube-api-access-vrlvs\") pod \"sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.713225 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.713163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:02:57.997965 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:57.997941    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"]
Sep 10 21:02:57.999970 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:02:57.999941    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8b545ed_1470_4694_a100_079996b76069.slice/crio-24eec0f7ebf9504e7c599efbb94dbf4f4f63dc58d3207f5632ecfe640b499564 WatchSource:0}: Error finding container 24eec0f7ebf9504e7c599efbb94dbf4f4f63dc58d3207f5632ecfe640b499564: Status 404 returned error can't find the container with id 24eec0f7ebf9504e7c599efbb94dbf4f4f63dc58d3207f5632ecfe640b499564
Sep 10 21:02:58.350864 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:58.350822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd" event={"ID":"b02e728a-ac8a-435b-b13c-55b836010998","Type":"ContainerStarted","Data":"7671082f9d8b36a5fd4496d10e0b2238c92dd15d835f45626186f7c6f9a20883"}
Sep 10 21:02:58.351035 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:58.350887    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:02:58.351035 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:58.350909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd" event={"ID":"b02e728a-ac8a-435b-b13c-55b836010998","Type":"ContainerStarted","Data":"d58835e0caf74475001e164739c993b97b50f51f3e486f11da354b33f6275a9c"}
Sep 10 21:02:58.352699 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:58.352665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr" event={"ID":"f8b545ed-1470-4694-a100-079996b76069","Type":"ContainerStarted","Data":"494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690"}
Sep 10 21:02:58.352800 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:58.352720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr" event={"ID":"f8b545ed-1470-4694-a100-079996b76069","Type":"ContainerStarted","Data":"24eec0f7ebf9504e7c599efbb94dbf4f4f63dc58d3207f5632ecfe640b499564"}
Sep 10 21:02:58.382507 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:58.382460    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd" podStartSLOduration=1.960015778 podStartE2EDuration="2.382446518s" totalImagesPullingTime="422.43074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:58.38062483 +0000 UTC m=+1614.291662104" watchObservedRunningTime="2026-09-10 21:02:58.382446518 +0000 UTC m=+1614.293483802"
Sep 10 21:02:58.405736 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:02:58.405680    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr" podStartSLOduration=1.405669001 podStartE2EDuration="1.405669001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:58.403941598 +0000 UTC m=+1614.314978882" watchObservedRunningTime="2026-09-10 21:02:58.405669001 +0000 UTC m=+1614.316706285"
Sep 10 21:03:02.725879 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:02.725842    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:03:04.366069 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:04.366041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:03:07.714256 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:07.714224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:03:07.714256 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:07.714266    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:03:07.727156 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:07.727129    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:03:08.421179 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:08.421151    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:03:25.202061 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.202029    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"]
Sep 10 21:03:25.206140 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.206116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.209429 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.209407    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-849fe6-serving-cert\""
Sep 10 21:03:25.227506 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.227477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"]
Sep 10 21:03:25.319394 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.319363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b678d9d-531c-4f94-ba36-d4c05dfb068f-openshift-service-ca\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.319550 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.319427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp44t\" (UniqueName: \"kubernetes.io/projected/3b678d9d-531c-4f94-ba36-d4c05dfb068f-kube-api-access-vp44t\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.319550 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.319488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.319550 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.319548    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-metrics-certs\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.420553 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.420520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.420773 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.420575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-metrics-certs\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.420773 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.420604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b678d9d-531c-4f94-ba36-d4c05dfb068f-openshift-service-ca\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.420773 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.420643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp44t\" (UniqueName: \"kubernetes.io/projected/3b678d9d-531c-4f94-ba36-d4c05dfb068f-kube-api-access-vp44t\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.420773 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:03:25.420705    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-serving-cert: secret "maas-api-e2e-api-b-849fe6-serving-cert" not found
Sep 10 21:03:25.421017 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:03:25.420792    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls podName:3b678d9d-531c-4f94-ba36-d4c05dfb068f nodeName:}" failed. No retries permitted until 2026-09-10 21:03:25.920771793 +0000 UTC m=+1641.831809057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls") pod "maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7" (UID: "3b678d9d-531c-4f94-ba36-d4c05dfb068f") : secret "maas-api-e2e-api-b-849fe6-serving-cert" not found
Sep 10 21:03:25.421239 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.421218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b678d9d-531c-4f94-ba36-d4c05dfb068f-openshift-service-ca\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.423101 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.423083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-metrics-certs\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.434618 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.434587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp44t\" (UniqueName: \"kubernetes.io/projected/3b678d9d-531c-4f94-ba36-d4c05dfb068f-kube-api-access-vp44t\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.925753 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.925716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:25.927928 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:25.927900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls\") pod \"maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:26.116204 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:26.116166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:26.390660 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:26.390629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"]
Sep 10 21:03:26.391960 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:03:26.391930    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b678d9d_531c_4f94_ba36_d4c05dfb068f.slice/crio-d9fb8fa4d5e1719aff0e7b5472dae0b575c74117f4728ed59cc22109f38957cf WatchSource:0}: Error finding container d9fb8fa4d5e1719aff0e7b5472dae0b575c74117f4728ed59cc22109f38957cf: Status 404 returned error can't find the container with id d9fb8fa4d5e1719aff0e7b5472dae0b575c74117f4728ed59cc22109f38957cf
Sep 10 21:03:26.518060 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:26.518029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7" event={"ID":"3b678d9d-531c-4f94-ba36-d4c05dfb068f","Type":"ContainerStarted","Data":"d9fb8fa4d5e1719aff0e7b5472dae0b575c74117f4728ed59cc22109f38957cf"}
Sep 10 21:03:27.527300 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:27.527250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7" event={"ID":"3b678d9d-531c-4f94-ba36-d4c05dfb068f","Type":"ContainerStarted","Data":"18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009"}
Sep 10 21:03:27.527725 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:27.527430    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:27.555526 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:27.555479    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7" podStartSLOduration=2.104743323 podStartE2EDuration="2.555463967s" totalImagesPullingTime="450.720644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:27.551938328 +0000 UTC m=+1643.462975611" watchObservedRunningTime="2026-09-10 21:03:27.555463967 +0000 UTC m=+1643.466501251"
Sep 10 21:03:33.539264 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:33.539228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:03:34.423415 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:34.423380    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:03:48.911097 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:03:48.911063    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:04:04.893287 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:04.893254    2565 scope.go:117] "RemoveContainer" containerID="0061dd21a042cd5138c5fb897d8b2bd6e01ea9d1c5f8abb6bbd146629055d305"
Sep 10 21:04:04.901905 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:04.901884    2565 scope.go:117] "RemoveContainer" containerID="c11c8d14509cf7bb496a01be18c059910c372951ed08e3d9dfa93a78de579a38"
Sep 10 21:04:05.601801 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:05.601758    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:04:07.512793 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.512761    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"]
Sep 10 21:04:07.513235 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.512993    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd" podUID="b02e728a-ac8a-435b-b13c-55b836010998" containerName="maas-api" containerID="cri-o://7671082f9d8b36a5fd4496d10e0b2238c92dd15d835f45626186f7c6f9a20883" gracePeriod=30
Sep 10 21:04:07.756259 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.756222    2565 generic.go:356] "Generic (PLEG): container finished" podID="b02e728a-ac8a-435b-b13c-55b836010998" containerID="7671082f9d8b36a5fd4496d10e0b2238c92dd15d835f45626186f7c6f9a20883" exitCode=0
Sep 10 21:04:07.756405 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.756293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd" event={"ID":"b02e728a-ac8a-435b-b13c-55b836010998","Type":"ContainerDied","Data":"7671082f9d8b36a5fd4496d10e0b2238c92dd15d835f45626186f7c6f9a20883"}
Sep 10 21:04:07.756405 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.756370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd" event={"ID":"b02e728a-ac8a-435b-b13c-55b836010998","Type":"ContainerDied","Data":"d58835e0caf74475001e164739c993b97b50f51f3e486f11da354b33f6275a9c"}
Sep 10 21:04:07.756405 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.756391    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d58835e0caf74475001e164739c993b97b50f51f3e486f11da354b33f6275a9c"
Sep 10 21:04:07.761358 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.761335    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:04:07.812795 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.812721    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls\") pod \"b02e728a-ac8a-435b-b13c-55b836010998\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") "
Sep 10 21:04:07.812931 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.812817    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pplt7\" (UniqueName: \"kubernetes.io/projected/b02e728a-ac8a-435b-b13c-55b836010998-kube-api-access-pplt7\") pod \"b02e728a-ac8a-435b-b13c-55b836010998\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") "
Sep 10 21:04:07.812931 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.812845    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-metrics-certs\") pod \"b02e728a-ac8a-435b-b13c-55b836010998\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") "
Sep 10 21:04:07.812931 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.812881    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e728a-ac8a-435b-b13c-55b836010998-openshift-service-ca\") pod \"b02e728a-ac8a-435b-b13c-55b836010998\" (UID: \"b02e728a-ac8a-435b-b13c-55b836010998\") "
Sep 10 21:04:07.813347 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.813315    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02e728a-ac8a-435b-b13c-55b836010998-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b02e728a-ac8a-435b-b13c-55b836010998" (UID: "b02e728a-ac8a-435b-b13c-55b836010998"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:07.814809 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.814785    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b02e728a-ac8a-435b-b13c-55b836010998" (UID: "b02e728a-ac8a-435b-b13c-55b836010998"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:07.814928 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.814904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b02e728a-ac8a-435b-b13c-55b836010998" (UID: "b02e728a-ac8a-435b-b13c-55b836010998"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:07.815017 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.814999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02e728a-ac8a-435b-b13c-55b836010998-kube-api-access-pplt7" (OuterVolumeSpecName: "kube-api-access-pplt7") pod "b02e728a-ac8a-435b-b13c-55b836010998" (UID: "b02e728a-ac8a-435b-b13c-55b836010998"). InnerVolumeSpecName "kube-api-access-pplt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:07.914733 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.914677    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:07.914733 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.914732    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplt7\" (UniqueName: \"kubernetes.io/projected/b02e728a-ac8a-435b-b13c-55b836010998-kube-api-access-pplt7\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:07.914937 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.914742    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e728a-ac8a-435b-b13c-55b836010998-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:07.914937 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:07.914751    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e728a-ac8a-435b-b13c-55b836010998-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:08.762307 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:08.762275    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"
Sep 10 21:04:08.780677 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:08.780650    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"]
Sep 10 21:04:08.785790 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:08.785759    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d66961-5cf5866f97-57lrd"]
Sep 10 21:04:10.591037 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:10.591000    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02e728a-ac8a-435b-b13c-55b836010998" path="/var/lib/kubelet/pods/b02e728a-ac8a-435b-b13c-55b836010998/volumes"
Sep 10 21:04:17.132206 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.132176    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8020-5ec550"]
Sep 10 21:04:17.132645 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.132632    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02e728a-ac8a-435b-b13c-55b836010998" containerName="maas-api"
Sep 10 21:04:17.132645 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.132647    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02e728a-ac8a-435b-b13c-55b836010998" containerName="maas-api"
Sep 10 21:04:17.132802 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.132790    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02e728a-ac8a-435b-b13c-55b836010998" containerName="maas-api"
Sep 10 21:04:17.136301 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.136279    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5ec550"
Sep 10 21:04:17.139451 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.139430    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-64jfg\""
Sep 10 21:04:17.144282 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.144254    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8020-5ec550"]
Sep 10 21:04:17.201542 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.201516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcd5g\" (UniqueName: \"kubernetes.io/projected/e1ed3c02-0ba4-4597-95e1-73aa5181036a-kube-api-access-pcd5g\") pod \"test-curl-8020-5ec550\" (UID: \"e1ed3c02-0ba4-4597-95e1-73aa5181036a\") " pod="opendatahub/test-curl-8020-5ec550"
Sep 10 21:04:17.302382 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.302349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcd5g\" (UniqueName: \"kubernetes.io/projected/e1ed3c02-0ba4-4597-95e1-73aa5181036a-kube-api-access-pcd5g\") pod \"test-curl-8020-5ec550\" (UID: \"e1ed3c02-0ba4-4597-95e1-73aa5181036a\") " pod="opendatahub/test-curl-8020-5ec550"
Sep 10 21:04:17.312586 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.312556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcd5g\" (UniqueName: \"kubernetes.io/projected/e1ed3c02-0ba4-4597-95e1-73aa5181036a-kube-api-access-pcd5g\") pod \"test-curl-8020-5ec550\" (UID: \"e1ed3c02-0ba4-4597-95e1-73aa5181036a\") " pod="opendatahub/test-curl-8020-5ec550"
Sep 10 21:04:17.447792 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.447713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5ec550"
Sep 10 21:04:17.718666 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.718623    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8020-5ec550"]
Sep 10 21:04:17.720978 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:04:17.720947    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1ed3c02_0ba4_4597_95e1_73aa5181036a.slice/crio-d9e8e1811f99b097787d1c8eae3015b9c4931f2d12bbd9009cdae030a4e723e3 WatchSource:0}: Error finding container d9e8e1811f99b097787d1c8eae3015b9c4931f2d12bbd9009cdae030a4e723e3: Status 404 returned error can't find the container with id d9e8e1811f99b097787d1c8eae3015b9c4931f2d12bbd9009cdae030a4e723e3
Sep 10 21:04:17.813521 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.813490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8020-5ec550" event={"ID":"e1ed3c02-0ba4-4597-95e1-73aa5181036a","Type":"ContainerStarted","Data":"703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73"}
Sep 10 21:04:17.813627 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.813532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8020-5ec550" event={"ID":"e1ed3c02-0ba4-4597-95e1-73aa5181036a","Type":"ContainerStarted","Data":"d9e8e1811f99b097787d1c8eae3015b9c4931f2d12bbd9009cdae030a4e723e3"}
Sep 10 21:04:17.832093 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:17.832049    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8020-5ec550" podStartSLOduration=0.832032988 podStartE2EDuration="832.032988ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:17.829853757 +0000 UTC m=+1693.740891041" watchObservedRunningTime="2026-09-10 21:04:17.832032988 +0000 UTC m=+1693.743070271"
Sep 10 21:04:18.428611 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.428574    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8020-5ec550"]
Sep 10 21:04:18.433659 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.433631    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8020-5ec550"]
Sep 10 21:04:18.557643 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.557614    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8020-7575de"]
Sep 10 21:04:18.558127 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.558114    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ed3c02-0ba4-4597-95e1-73aa5181036a" containerName="test-curl-8020-5ec550"
Sep 10 21:04:18.558180 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.558129    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ed3c02-0ba4-4597-95e1-73aa5181036a" containerName="test-curl-8020-5ec550"
Sep 10 21:04:18.558227 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.558218    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ed3c02-0ba4-4597-95e1-73aa5181036a" containerName="test-curl-8020-5ec550"
Sep 10 21:04:18.561312 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.561296    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-7575de"
Sep 10 21:04:18.568002 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.567978    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8020-7575de"]
Sep 10 21:04:18.614343 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.614315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dldf\" (UniqueName: \"kubernetes.io/projected/7f69115f-3d6e-4d57-ad02-42d8015b365e-kube-api-access-6dldf\") pod \"test-curl-8020-7575de\" (UID: \"7f69115f-3d6e-4d57-ad02-42d8015b365e\") " pod="opendatahub/test-curl-8020-7575de"
Sep 10 21:04:18.715923 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.715842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dldf\" (UniqueName: \"kubernetes.io/projected/7f69115f-3d6e-4d57-ad02-42d8015b365e-kube-api-access-6dldf\") pod \"test-curl-8020-7575de\" (UID: \"7f69115f-3d6e-4d57-ad02-42d8015b365e\") " pod="opendatahub/test-curl-8020-7575de"
Sep 10 21:04:18.725419 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.725395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dldf\" (UniqueName: \"kubernetes.io/projected/7f69115f-3d6e-4d57-ad02-42d8015b365e-kube-api-access-6dldf\") pod \"test-curl-8020-7575de\" (UID: \"7f69115f-3d6e-4d57-ad02-42d8015b365e\") " pod="opendatahub/test-curl-8020-7575de"
Sep 10 21:04:18.819099 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:04:18.819074    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8020-5ec550" container="test-curl-8020-5ec550"
Sep 10 21:04:18.872409 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:18.872390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-7575de"
Sep 10 21:04:19.141965 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:19.141939    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8020-7575de"]
Sep 10 21:04:19.144304 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:04:19.144273    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f69115f_3d6e_4d57_ad02_42d8015b365e.slice/crio-b262c5be789e2f7dd3e26ef9d9f6ec0b5eecadd48c55071e40213fea40a9cc9c WatchSource:0}: Error finding container b262c5be789e2f7dd3e26ef9d9f6ec0b5eecadd48c55071e40213fea40a9cc9c: Status 404 returned error can't find the container with id b262c5be789e2f7dd3e26ef9d9f6ec0b5eecadd48c55071e40213fea40a9cc9c
Sep 10 21:04:19.825923 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:19.825880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8020-7575de" event={"ID":"7f69115f-3d6e-4d57-ad02-42d8015b365e","Type":"ContainerStarted","Data":"3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06"}
Sep 10 21:04:19.825923 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:19.825927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8020-7575de" event={"ID":"7f69115f-3d6e-4d57-ad02-42d8015b365e","Type":"ContainerStarted","Data":"b262c5be789e2f7dd3e26ef9d9f6ec0b5eecadd48c55071e40213fea40a9cc9c"}
Sep 10 21:04:19.826406 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:19.825975    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8020-5ec550" podUID="e1ed3c02-0ba4-4597-95e1-73aa5181036a" containerName="test-curl-8020-5ec550" containerID="cri-o://703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73" gracePeriod=2
Sep 10 21:04:19.846011 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:19.845965    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8020-7575de" podStartSLOduration=1.845950648 podStartE2EDuration="1.845950648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:19.844093598 +0000 UTC m=+1695.755130881" watchObservedRunningTime="2026-09-10 21:04:19.845950648 +0000 UTC m=+1695.756987931"
Sep 10 21:04:20.470866 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:20.470833    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8020-7575de"]
Sep 10 21:04:20.474933 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:20.474907    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8020-7575de"]
Sep 10 21:04:20.832817 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:04:20.832786    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8020-7575de" container="test-curl-8020-7575de"
Sep 10 21:04:21.839964 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:21.839928    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8020-7575de" podUID="7f69115f-3d6e-4d57-ad02-42d8015b365e" containerName="test-curl-8020-7575de" containerID="cri-o://3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06" gracePeriod=2
Sep 10 21:04:21.996099 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:21.996076    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5ec550"
Sep 10 21:04:22.141009 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.140975    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcd5g\" (UniqueName: \"kubernetes.io/projected/e1ed3c02-0ba4-4597-95e1-73aa5181036a-kube-api-access-pcd5g\") pod \"e1ed3c02-0ba4-4597-95e1-73aa5181036a\" (UID: \"e1ed3c02-0ba4-4597-95e1-73aa5181036a\") "
Sep 10 21:04:22.142963 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.142927    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ed3c02-0ba4-4597-95e1-73aa5181036a-kube-api-access-pcd5g" (OuterVolumeSpecName: "kube-api-access-pcd5g") pod "e1ed3c02-0ba4-4597-95e1-73aa5181036a" (UID: "e1ed3c02-0ba4-4597-95e1-73aa5181036a"). InnerVolumeSpecName "kube-api-access-pcd5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:22.242035 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.242011    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcd5g\" (UniqueName: \"kubernetes.io/projected/e1ed3c02-0ba4-4597-95e1-73aa5181036a-kube-api-access-pcd5g\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:22.509277 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.509204    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"]
Sep 10 21:04:22.592230 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.592200    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ed3c02-0ba4-4597-95e1-73aa5181036a" path="/var/lib/kubelet/pods/e1ed3c02-0ba4-4597-95e1-73aa5181036a/volumes"
Sep 10 21:04:22.847221 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.847191    2565 generic.go:356] "Generic (PLEG): container finished" podID="e1ed3c02-0ba4-4597-95e1-73aa5181036a" containerID="703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73" exitCode=137
Sep 10 21:04:22.847589 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.847228    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-5ec550"
Sep 10 21:04:22.847589 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.847284    2565 scope.go:117] "RemoveContainer" containerID="703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73"
Sep 10 21:04:22.856437 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.856415    2565 scope.go:117] "RemoveContainer" containerID="703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73"
Sep 10 21:04:22.856767 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:04:22.856748    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73\": container with ID starting with 703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73 not found: ID does not exist" containerID="703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73"
Sep 10 21:04:22.856840 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:22.856776    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73"} err="failed to get container status \"703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73\": rpc error: code = NotFound desc = could not find container \"703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73\": container with ID starting with 703fb37b43d36a8f9683cceee99959902242f18bcda331c671ccbbf407110d73 not found: ID does not exist"
Sep 10 21:04:23.997055 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:23.997032    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-7575de"
Sep 10 21:04:24.058006 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.057978    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dldf\" (UniqueName: \"kubernetes.io/projected/7f69115f-3d6e-4d57-ad02-42d8015b365e-kube-api-access-6dldf\") pod \"7f69115f-3d6e-4d57-ad02-42d8015b365e\" (UID: \"7f69115f-3d6e-4d57-ad02-42d8015b365e\") "
Sep 10 21:04:24.059931 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.059909    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f69115f-3d6e-4d57-ad02-42d8015b365e-kube-api-access-6dldf" (OuterVolumeSpecName: "kube-api-access-6dldf") pod "7f69115f-3d6e-4d57-ad02-42d8015b365e" (UID: "7f69115f-3d6e-4d57-ad02-42d8015b365e"). InnerVolumeSpecName "kube-api-access-6dldf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:24.159001 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.158938    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dldf\" (UniqueName: \"kubernetes.io/projected/7f69115f-3d6e-4d57-ad02-42d8015b365e-kube-api-access-6dldf\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:24.299154 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.299121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:04:24.591487 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.591462    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f69115f-3d6e-4d57-ad02-42d8015b365e" path="/var/lib/kubelet/pods/7f69115f-3d6e-4d57-ad02-42d8015b365e/volumes"
Sep 10 21:04:24.862104 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.862016    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f69115f-3d6e-4d57-ad02-42d8015b365e" containerID="3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06" exitCode=137
Sep 10 21:04:24.862104 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.862065    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8020-7575de"
Sep 10 21:04:24.862307 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.862124    2565 scope.go:117] "RemoveContainer" containerID="3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06"
Sep 10 21:04:24.870938 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.870924    2565 scope.go:117] "RemoveContainer" containerID="3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06"
Sep 10 21:04:24.871195 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:04:24.871176    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06\": container with ID starting with 3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06 not found: ID does not exist" containerID="3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06"
Sep 10 21:04:24.871244 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:24.871203    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06"} err="failed to get container status \"3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06\": rpc error: code = NotFound desc = could not find container \"3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06\": container with ID starting with 3c8cc8864bdc87d8c479b5595fa49321e0c16592c3ad091c0c664100c63c5d06 not found: ID does not exist"
Sep 10 21:04:30.963000 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:30.962927    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"]
Sep 10 21:04:30.963377 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:30.963195    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7" podUID="3b678d9d-531c-4f94-ba36-d4c05dfb068f" containerName="maas-api" containerID="cri-o://18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009" gracePeriod=30
Sep 10 21:04:31.217889 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.217834    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:04:31.328177 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.328138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls\") pod \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") "
Sep 10 21:04:31.328331 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.328235    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp44t\" (UniqueName: \"kubernetes.io/projected/3b678d9d-531c-4f94-ba36-d4c05dfb068f-kube-api-access-vp44t\") pod \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") "
Sep 10 21:04:31.328395 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.328380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b678d9d-531c-4f94-ba36-d4c05dfb068f-openshift-service-ca\") pod \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") "
Sep 10 21:04:31.328431 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.328416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-metrics-certs\") pod \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\" (UID: \"3b678d9d-531c-4f94-ba36-d4c05dfb068f\") "
Sep 10 21:04:31.328789 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.328742    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b678d9d-531c-4f94-ba36-d4c05dfb068f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b678d9d-531c-4f94-ba36-d4c05dfb068f" (UID: "3b678d9d-531c-4f94-ba36-d4c05dfb068f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:31.330385 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.330349    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b678d9d-531c-4f94-ba36-d4c05dfb068f" (UID: "3b678d9d-531c-4f94-ba36-d4c05dfb068f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:31.330633 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.330598    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b678d9d-531c-4f94-ba36-d4c05dfb068f" (UID: "3b678d9d-531c-4f94-ba36-d4c05dfb068f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:31.330753 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.330657    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b678d9d-531c-4f94-ba36-d4c05dfb068f-kube-api-access-vp44t" (OuterVolumeSpecName: "kube-api-access-vp44t") pod "3b678d9d-531c-4f94-ba36-d4c05dfb068f" (UID: "3b678d9d-531c-4f94-ba36-d4c05dfb068f"). InnerVolumeSpecName "kube-api-access-vp44t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:31.429892 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.429859    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp44t\" (UniqueName: \"kubernetes.io/projected/3b678d9d-531c-4f94-ba36-d4c05dfb068f-kube-api-access-vp44t\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.429892 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.429887    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b678d9d-531c-4f94-ba36-d4c05dfb068f-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.429892 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.429897    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.430103 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.429905    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b678d9d-531c-4f94-ba36-d4c05dfb068f-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.902278 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.902247    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b678d9d-531c-4f94-ba36-d4c05dfb068f" containerID="18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009" exitCode=0
Sep 10 21:04:31.902440 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.902293    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"
Sep 10 21:04:31.902440 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.902341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7" event={"ID":"3b678d9d-531c-4f94-ba36-d4c05dfb068f","Type":"ContainerDied","Data":"18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009"}
Sep 10 21:04:31.902440 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.902393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7" event={"ID":"3b678d9d-531c-4f94-ba36-d4c05dfb068f","Type":"ContainerDied","Data":"d9fb8fa4d5e1719aff0e7b5472dae0b575c74117f4728ed59cc22109f38957cf"}
Sep 10 21:04:31.902440 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.902415    2565 scope.go:117] "RemoveContainer" containerID="18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009"
Sep 10 21:04:31.911417 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.911400    2565 scope.go:117] "RemoveContainer" containerID="18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009"
Sep 10 21:04:31.911674 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:04:31.911651    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009\": container with ID starting with 18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009 not found: ID does not exist" containerID="18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009"
Sep 10 21:04:31.911794 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.911749    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009"} err="failed to get container status \"18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009\": rpc error: code = NotFound desc = could not find container \"18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009\": container with ID starting with 18702ea44a5cb27549d436cf4783ce439252621b7921db1c3002b800be41a009 not found: ID does not exist"
Sep 10 21:04:31.925755 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.925734    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"]
Sep 10 21:04:31.930170 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:31.930150    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-849fe6-7bdd59fd5d-4h7v7"]
Sep 10 21:04:32.591925 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:32.591890    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b678d9d-531c-4f94-ba36-d4c05dfb068f" path="/var/lib/kubelet/pods/3b678d9d-531c-4f94-ba36-d4c05dfb068f/volumes"
Sep 10 21:04:37.528332 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.528290    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr" podUID="f8b545ed-1470-4694-a100-079996b76069" containerName="main" containerID="cri-o://494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690" gracePeriod=45
Sep 10 21:04:37.783593 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.783534    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:04:37.886190 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886156    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrlvs\" (UniqueName: \"kubernetes.io/projected/f8b545ed-1470-4694-a100-079996b76069-kube-api-access-vrlvs\") pod \"f8b545ed-1470-4694-a100-079996b76069\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") "
Sep 10 21:04:37.886359 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886197    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b545ed-1470-4694-a100-079996b76069-tls-certs\") pod \"f8b545ed-1470-4694-a100-079996b76069\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") "
Sep 10 21:04:37.886359 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-home\") pod \"f8b545ed-1470-4694-a100-079996b76069\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") "
Sep 10 21:04:37.886359 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886351    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-model-cache\") pod \"f8b545ed-1470-4694-a100-079996b76069\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") "
Sep 10 21:04:37.886528 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-tmp-dir\") pod \"f8b545ed-1470-4694-a100-079996b76069\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") "
Sep 10 21:04:37.886528 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-dshm\") pod \"f8b545ed-1470-4694-a100-079996b76069\" (UID: \"f8b545ed-1470-4694-a100-079996b76069\") "
Sep 10 21:04:37.886647 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-home" (OuterVolumeSpecName: "home") pod "f8b545ed-1470-4694-a100-079996b76069" (UID: "f8b545ed-1470-4694-a100-079996b76069"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:37.886728 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8b545ed-1470-4694-a100-079996b76069" (UID: "f8b545ed-1470-4694-a100-079996b76069"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:37.886728 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8b545ed-1470-4694-a100-079996b76069" (UID: "f8b545ed-1470-4694-a100-079996b76069"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:37.886856 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886839    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:37.886913 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886859    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:37.886913 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.886867    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:37.888460 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.888439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b545ed-1470-4694-a100-079996b76069-kube-api-access-vrlvs" (OuterVolumeSpecName: "kube-api-access-vrlvs") pod "f8b545ed-1470-4694-a100-079996b76069" (UID: "f8b545ed-1470-4694-a100-079996b76069"). InnerVolumeSpecName "kube-api-access-vrlvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:37.888752 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.888726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b545ed-1470-4694-a100-079996b76069-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8b545ed-1470-4694-a100-079996b76069" (UID: "f8b545ed-1470-4694-a100-079996b76069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:37.888752 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.888741    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-dshm" (OuterVolumeSpecName: "dshm") pod "f8b545ed-1470-4694-a100-079996b76069" (UID: "f8b545ed-1470-4694-a100-079996b76069"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:37.940913 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.940876    2565 generic.go:356] "Generic (PLEG): container finished" podID="f8b545ed-1470-4694-a100-079996b76069" containerID="494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690" exitCode=0
Sep 10 21:04:37.941099 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.940947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr" event={"ID":"f8b545ed-1470-4694-a100-079996b76069","Type":"ContainerDied","Data":"494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690"}
Sep 10 21:04:37.941099 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.941002    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr" event={"ID":"f8b545ed-1470-4694-a100-079996b76069","Type":"ContainerDied","Data":"24eec0f7ebf9504e7c599efbb94dbf4f4f63dc58d3207f5632ecfe640b499564"}
Sep 10 21:04:37.941099 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.941001    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"
Sep 10 21:04:37.941099 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.941018    2565 scope.go:117] "RemoveContainer" containerID="494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690"
Sep 10 21:04:37.950745 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.950730    2565 scope.go:117] "RemoveContainer" containerID="494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690"
Sep 10 21:04:37.951036 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:04:37.951017    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690\": container with ID starting with 494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690 not found: ID does not exist" containerID="494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690"
Sep 10 21:04:37.951110 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.951043    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690"} err="failed to get container status \"494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690\": rpc error: code = NotFound desc = could not find container \"494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690\": container with ID starting with 494591009a99fc5f9f5b9fa4a9dfbc0dcf4629bc966646bdf6f911bc1fe8d690 not found: ID does not exist"
Sep 10 21:04:37.964828 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.964809    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"]
Sep 10 21:04:37.970861 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.970840    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/sub-test-model-3fc2b8-kserve-5b88fdbb67-9q4pr"]
Sep 10 21:04:37.988148 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.988130    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrlvs\" (UniqueName: \"kubernetes.io/projected/f8b545ed-1470-4694-a100-079996b76069-kube-api-access-vrlvs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:37.988217 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.988150    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b545ed-1470-4694-a100-079996b76069-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:37.988217 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:37.988160    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b545ed-1470-4694-a100-079996b76069-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:38.593616 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:38.593579    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b545ed-1470-4694-a100-079996b76069" path="/var/lib/kubelet/pods/f8b545ed-1470-4694-a100-079996b76069/volumes"
Sep 10 21:04:43.200938 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:43.200902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:04:48.829903 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:48.829868    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"]
Sep 10 21:04:48.830380 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:48.830114    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9" podUID="78f228c4-8364-46ef-93d1-78c5c4293ee2" containerName="maas-api" containerID="cri-o://08aaacba053f30bb53bb309a265dd5653c624ae71af337405451f2deafeb1d98" gracePeriod=30
Sep 10 21:04:49.004439 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.004400    2565 generic.go:356] "Generic (PLEG): container finished" podID="78f228c4-8364-46ef-93d1-78c5c4293ee2" containerID="08aaacba053f30bb53bb309a265dd5653c624ae71af337405451f2deafeb1d98" exitCode=0
Sep 10 21:04:49.004731 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.004447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9" event={"ID":"78f228c4-8364-46ef-93d1-78c5c4293ee2","Type":"ContainerDied","Data":"08aaacba053f30bb53bb309a265dd5653c624ae71af337405451f2deafeb1d98"}
Sep 10 21:04:49.106761 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.106736    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 21:04:49.184374 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.184347    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f228c4-8364-46ef-93d1-78c5c4293ee2-openshift-service-ca\") pod \"78f228c4-8364-46ef-93d1-78c5c4293ee2\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") "
Sep 10 21:04:49.184374 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.184399    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-maas-api-tls\") pod \"78f228c4-8364-46ef-93d1-78c5c4293ee2\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") "
Sep 10 21:04:49.184637 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.184447    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9svtt\" (UniqueName: \"kubernetes.io/projected/78f228c4-8364-46ef-93d1-78c5c4293ee2-kube-api-access-9svtt\") pod \"78f228c4-8364-46ef-93d1-78c5c4293ee2\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") "
Sep 10 21:04:49.184637 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.184489    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-metrics-certs\") pod \"78f228c4-8364-46ef-93d1-78c5c4293ee2\" (UID: \"78f228c4-8364-46ef-93d1-78c5c4293ee2\") "
Sep 10 21:04:49.184810 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.184740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f228c4-8364-46ef-93d1-78c5c4293ee2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78f228c4-8364-46ef-93d1-78c5c4293ee2" (UID: "78f228c4-8364-46ef-93d1-78c5c4293ee2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:49.184887 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.184867    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f228c4-8364-46ef-93d1-78c5c4293ee2-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:49.186826 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.186798    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78f228c4-8364-46ef-93d1-78c5c4293ee2" (UID: "78f228c4-8364-46ef-93d1-78c5c4293ee2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:49.186826 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.186809    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "78f228c4-8364-46ef-93d1-78c5c4293ee2" (UID: "78f228c4-8364-46ef-93d1-78c5c4293ee2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:49.187093 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.187077    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f228c4-8364-46ef-93d1-78c5c4293ee2-kube-api-access-9svtt" (OuterVolumeSpecName: "kube-api-access-9svtt") pod "78f228c4-8364-46ef-93d1-78c5c4293ee2" (UID: "78f228c4-8364-46ef-93d1-78c5c4293ee2"). InnerVolumeSpecName "kube-api-access-9svtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:49.285995 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.285954    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:49.285995 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.285989    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9svtt\" (UniqueName: \"kubernetes.io/projected/78f228c4-8364-46ef-93d1-78c5c4293ee2-kube-api-access-9svtt\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:49.285995 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:49.286001    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78f228c4-8364-46ef-93d1-78c5c4293ee2-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:04:50.011878 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:50.011851    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"
Sep 10 21:04:50.012271 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:50.011880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9" event={"ID":"78f228c4-8364-46ef-93d1-78c5c4293ee2","Type":"ContainerDied","Data":"f626ea317c4c59faa7e058bc0602e0fc03842040d84702e4c2679c6899b670d1"}
Sep 10 21:04:50.012271 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:50.011962    2565 scope.go:117] "RemoveContainer" containerID="08aaacba053f30bb53bb309a265dd5653c624ae71af337405451f2deafeb1d98"
Sep 10 21:04:50.037538 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:50.037509    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"]
Sep 10 21:04:50.041376 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:50.041356    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cd89ac-5f4ff4976b-jflf9"]
Sep 10 21:04:50.591998 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:50.591967    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f228c4-8364-46ef-93d1-78c5c4293ee2" path="/var/lib/kubelet/pods/78f228c4-8364-46ef-93d1-78c5c4293ee2/volumes"
Sep 10 21:04:52.005594 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:04:52.005554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:05:06.830324 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:06.830293    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"]
Sep 10 21:05:06.830733 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:06.830482    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2" podUID="f3cab697-3320-40b7-88a4-ed45a4016c86" containerName="maas-api" containerID="cri-o://8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c" gracePeriod=30
Sep 10 21:05:07.104419 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.104395    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 21:05:07.117471 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.117440    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3cab697-3320-40b7-88a4-ed45a4016c86" containerID="8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c" exitCode=0
Sep 10 21:05:07.117596 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.117507    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"
Sep 10 21:05:07.117596 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.117502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2" event={"ID":"f3cab697-3320-40b7-88a4-ed45a4016c86","Type":"ContainerDied","Data":"8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c"}
Sep 10 21:05:07.117709 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.117655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2" event={"ID":"f3cab697-3320-40b7-88a4-ed45a4016c86","Type":"ContainerDied","Data":"c4a77267a796787a7539bf95a3301f663b98aebfb4ea78d5b4fbf023cb3caf1c"}
Sep 10 21:05:07.117709 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.117679    2565 scope.go:117] "RemoveContainer" containerID="8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c"
Sep 10 21:05:07.126553 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.126535    2565 scope.go:117] "RemoveContainer" containerID="8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c"
Sep 10 21:05:07.126831 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:05:07.126809    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c\": container with ID starting with 8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c not found: ID does not exist" containerID="8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c"
Sep 10 21:05:07.126899 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.126843    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c"} err="failed to get container status \"8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c\": rpc error: code = NotFound desc = could not find container \"8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c\": container with ID starting with 8daede1788744f2f134dba8d8cee73bdf626c0a6740e69372801f687a3da756c not found: ID does not exist"
Sep 10 21:05:07.253042 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.253012    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-maas-api-tls\") pod \"f3cab697-3320-40b7-88a4-ed45a4016c86\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") "
Sep 10 21:05:07.253175 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.253080    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-metrics-certs\") pod \"f3cab697-3320-40b7-88a4-ed45a4016c86\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") "
Sep 10 21:05:07.253175 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.253121    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cab697-3320-40b7-88a4-ed45a4016c86-openshift-service-ca\") pod \"f3cab697-3320-40b7-88a4-ed45a4016c86\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") "
Sep 10 21:05:07.253274 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.253186    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k458g\" (UniqueName: \"kubernetes.io/projected/f3cab697-3320-40b7-88a4-ed45a4016c86-kube-api-access-k458g\") pod \"f3cab697-3320-40b7-88a4-ed45a4016c86\" (UID: \"f3cab697-3320-40b7-88a4-ed45a4016c86\") "
Sep 10 21:05:07.253569 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.253536    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cab697-3320-40b7-88a4-ed45a4016c86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3cab697-3320-40b7-88a4-ed45a4016c86" (UID: "f3cab697-3320-40b7-88a4-ed45a4016c86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:05:07.255219 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.255190    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3cab697-3320-40b7-88a4-ed45a4016c86" (UID: "f3cab697-3320-40b7-88a4-ed45a4016c86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:07.255331 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.255235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3cab697-3320-40b7-88a4-ed45a4016c86" (UID: "f3cab697-3320-40b7-88a4-ed45a4016c86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:07.255331 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.255249    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cab697-3320-40b7-88a4-ed45a4016c86-kube-api-access-k458g" (OuterVolumeSpecName: "kube-api-access-k458g") pod "f3cab697-3320-40b7-88a4-ed45a4016c86" (UID: "f3cab697-3320-40b7-88a4-ed45a4016c86"). InnerVolumeSpecName "kube-api-access-k458g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:07.354206 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.354176    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k458g\" (UniqueName: \"kubernetes.io/projected/f3cab697-3320-40b7-88a4-ed45a4016c86-kube-api-access-k458g\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:05:07.354206 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.354204    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:05:07.354353 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.354214    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cab697-3320-40b7-88a4-ed45a4016c86-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:05:07.354353 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.354222    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cab697-3320-40b7-88a4-ed45a4016c86-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:05:07.442434 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.442410    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"]
Sep 10 21:05:07.449870 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:07.449850    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bbb40-7dc9f855db-lrhb2"]
Sep 10 21:05:08.410489 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:08.410450    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:05:08.592187 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:08.592154    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3cab697-3320-40b7-88a4-ed45a4016c86" path="/var/lib/kubelet/pods/f3cab697-3320-40b7-88a4-ed45a4016c86/volumes"
Sep 10 21:05:18.092055 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:18.092019    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:05:27.195411 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:27.195381    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:05:37.897288 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:37.897253    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:05:47.016571 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.016536    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:05:47.879233 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879203    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"]
Sep 10 21:05:47.879628 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879616    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f228c4-8364-46ef-93d1-78c5c4293ee2" containerName="maas-api"
Sep 10 21:05:47.879673 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879629    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f228c4-8364-46ef-93d1-78c5c4293ee2" containerName="maas-api"
Sep 10 21:05:47.879673 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879646    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b545ed-1470-4694-a100-079996b76069" containerName="main"
Sep 10 21:05:47.879673 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879652    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b545ed-1470-4694-a100-079996b76069" containerName="main"
Sep 10 21:05:47.879673 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879663    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f69115f-3d6e-4d57-ad02-42d8015b365e" containerName="test-curl-8020-7575de"
Sep 10 21:05:47.879673 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879670    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f69115f-3d6e-4d57-ad02-42d8015b365e" containerName="test-curl-8020-7575de"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879677    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cab697-3320-40b7-88a4-ed45a4016c86" containerName="maas-api"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879753    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cab697-3320-40b7-88a4-ed45a4016c86" containerName="maas-api"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879762    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b678d9d-531c-4f94-ba36-d4c05dfb068f" containerName="maas-api"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879767    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b678d9d-531c-4f94-ba36-d4c05dfb068f" containerName="maas-api"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879839    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f69115f-3d6e-4d57-ad02-42d8015b365e" containerName="test-curl-8020-7575de"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879847    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cab697-3320-40b7-88a4-ed45a4016c86" containerName="maas-api"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879856    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b678d9d-531c-4f94-ba36-d4c05dfb068f" containerName="maas-api"
Sep 10 21:05:47.879863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879864    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f228c4-8364-46ef-93d1-78c5c4293ee2" containerName="maas-api"
Sep 10 21:05:47.880110 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.879870    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b545ed-1470-4694-a100-079996b76069" containerName="main"
Sep 10 21:05:47.893662 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.893631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:47.896259 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.896231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"]
Sep 10 21:05:47.897122 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:47.897095    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b73da3-serving-cert\""
Sep 10 21:05:48.008203 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.008169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlhp\" (UniqueName: \"kubernetes.io/projected/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-kube-api-access-khlhp\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.008357 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.008215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.008357 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.008237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.008357 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.008301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.109176 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.109145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.109584 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.109239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlhp\" (UniqueName: \"kubernetes.io/projected/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-kube-api-access-khlhp\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.109584 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.109281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.109584 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.109310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.109584 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:05:48.109419    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-serving-cert: secret "maas-api-e2e-ipp-a-b73da3-serving-cert" not found
Sep 10 21:05:48.109584 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:05:48.109486    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls podName:8bd36734-a95a-4ac4-9a09-aec24cccb0b7 nodeName:}" failed. No retries permitted until 2026-09-10 21:05:48.609466103 +0000 UTC m=+1784.520503366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls") pod "maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q" (UID: "8bd36734-a95a-4ac4-9a09-aec24cccb0b7") : secret "maas-api-e2e-ipp-a-b73da3-serving-cert" not found
Sep 10 21:05:48.110066 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.110043    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.111696 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.111670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.119164 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.119137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlhp\" (UniqueName: \"kubernetes.io/projected/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-kube-api-access-khlhp\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.614067 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.614035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.616418 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.616395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:48.806340 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:48.806304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:49.080679 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:49.080651    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"]
Sep 10 21:05:49.082145 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:05:49.082122    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd36734_a95a_4ac4_9a09_aec24cccb0b7.slice/crio-04e05f61f52f86472b41ca1546f9d2b4ee766cf7e44215b43659f5257989e007 WatchSource:0}: Error finding container 04e05f61f52f86472b41ca1546f9d2b4ee766cf7e44215b43659f5257989e007: Status 404 returned error can't find the container with id 04e05f61f52f86472b41ca1546f9d2b4ee766cf7e44215b43659f5257989e007
Sep 10 21:05:49.352567 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:49.352533    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q" event={"ID":"8bd36734-a95a-4ac4-9a09-aec24cccb0b7","Type":"ContainerStarted","Data":"04e05f61f52f86472b41ca1546f9d2b4ee766cf7e44215b43659f5257989e007"}
Sep 10 21:05:50.359698 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:50.359659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q" event={"ID":"8bd36734-a95a-4ac4-9a09-aec24cccb0b7","Type":"ContainerStarted","Data":"56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d"}
Sep 10 21:05:50.360142 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:50.359734    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:50.380607 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:50.380560    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q" podStartSLOduration=2.8998283259999997 podStartE2EDuration="3.380544822s" totalImagesPullingTime="480.716496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:50.378712515 +0000 UTC m=+1786.289749793" watchObservedRunningTime="2026-09-10 21:05:50.380544822 +0000 UTC m=+1786.291582106"
Sep 10 21:05:56.371456 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:56.371424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:05:56.888436 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:05:56.888406    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:06:04.693849 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:04.693822    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:06:04.698201 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:04.698175    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:06:04.698345 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:04.698215    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:06:04.702094 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:04.702070    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:06:17.013710 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.013661    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"]
Sep 10 21:06:17.017360 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.017340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.020940 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.020914    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-18b9ac-serving-cert\""
Sep 10 21:06:17.036861 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.036836    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"]
Sep 10 21:06:17.173892 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.173861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-maas-api-tls\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.174069 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.173910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f5c866-ffc1-4731-b546-0222f3994056-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.174069 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.173932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvcl\" (UniqueName: \"kubernetes.io/projected/07f5c866-ffc1-4731-b546-0222f3994056-kube-api-access-swvcl\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.174069 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.173978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-metrics-certs\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.275218 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.275137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-maas-api-tls\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.275218 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.275185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f5c866-ffc1-4731-b546-0222f3994056-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.275390 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.275365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvcl\" (UniqueName: \"kubernetes.io/projected/07f5c866-ffc1-4731-b546-0222f3994056-kube-api-access-swvcl\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.275439 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.275399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-metrics-certs\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.275945 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.275924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f5c866-ffc1-4731-b546-0222f3994056-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.277907 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.277879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-metrics-certs\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.277989 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.277879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-maas-api-tls\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.284111 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.284094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvcl\" (UniqueName: \"kubernetes.io/projected/07f5c866-ffc1-4731-b546-0222f3994056-kube-api-access-swvcl\") pod \"maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.330623 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.330599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:17.603841 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:17.603811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"]
Sep 10 21:06:17.604583 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:06:17.604557    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f5c866_ffc1_4731_b546_0222f3994056.slice/crio-62a6c3abbf8f3850cb6cace4e9fa834c1098707baf367a41c918ba24cafa8bf0 WatchSource:0}: Error finding container 62a6c3abbf8f3850cb6cace4e9fa834c1098707baf367a41c918ba24cafa8bf0: Status 404 returned error can't find the container with id 62a6c3abbf8f3850cb6cace4e9fa834c1098707baf367a41c918ba24cafa8bf0
Sep 10 21:06:18.519888 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:18.519849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4" event={"ID":"07f5c866-ffc1-4731-b546-0222f3994056","Type":"ContainerStarted","Data":"dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6"}
Sep 10 21:06:18.520279 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:18.519905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4" event={"ID":"07f5c866-ffc1-4731-b546-0222f3994056","Type":"ContainerStarted","Data":"62a6c3abbf8f3850cb6cace4e9fa834c1098707baf367a41c918ba24cafa8bf0"}
Sep 10 21:06:18.520279 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:18.519936    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:18.539560 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:18.539505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4" podStartSLOduration=2.237034926 podStartE2EDuration="2.539486919s" totalImagesPullingTime="302.451993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:18.538474949 +0000 UTC m=+1814.449512258" watchObservedRunningTime="2026-09-10 21:06:18.539486919 +0000 UTC m=+1814.450524203"
Sep 10 21:06:23.692960 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:23.692924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:06:24.531613 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:24.531584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:06:45.811938 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:45.811904    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lq8k2_278ec5f3-c99d-47bb-99d1-0185806fe1e0/payload-processing/0.log"
Sep 10 21:06:56.797317 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:06:56.797284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:07:20.398728 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:20.398696    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:07:26.752010 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:26.751981    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lq8k2_278ec5f3-c99d-47bb-99d1-0185806fe1e0/payload-processing/0.log"
Sep 10 21:07:31.294586 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.294553    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"]
Sep 10 21:07:31.298748 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.298724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.302015 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.301997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-eb2a90-serving-cert\""
Sep 10 21:07:31.310720 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.310698    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"]
Sep 10 21:07:31.429984 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.429950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fdc9cd-5090-43b6-b765-f65f80219f4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.429984 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.429987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bhw\" (UniqueName: \"kubernetes.io/projected/46fdc9cd-5090-43b6-b765-f65f80219f4f-kube-api-access-m4bhw\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.430183 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.430014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.430238 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.430171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.531049 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.531004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fdc9cd-5090-43b6-b765-f65f80219f4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.531049 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.531051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bhw\" (UniqueName: \"kubernetes.io/projected/46fdc9cd-5090-43b6-b765-f65f80219f4f-kube-api-access-m4bhw\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.531303 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.531083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.531303 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.531117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.531303 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:07:31.531250    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-serving-cert: secret "maas-api-e2e-ipp-cleanup-eb2a90-serving-cert" not found
Sep 10 21:07:31.531477 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:07:31.531338    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls podName:46fdc9cd-5090-43b6-b765-f65f80219f4f nodeName:}" failed. No retries permitted until 2026-09-10 21:07:32.031316632 +0000 UTC m=+1887.942353901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526" (UID: "46fdc9cd-5090-43b6-b765-f65f80219f4f") : secret "maas-api-e2e-ipp-cleanup-eb2a90-serving-cert" not found
Sep 10 21:07:31.531673 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.531654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fdc9cd-5090-43b6-b765-f65f80219f4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.533546 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.533516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:31.545427 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:31.545369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bhw\" (UniqueName: \"kubernetes.io/projected/46fdc9cd-5090-43b6-b765-f65f80219f4f-kube-api-access-m4bhw\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:32.036291 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:32.036255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:32.038863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:32.038834    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:32.209661 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:32.209625    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:32.484273 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:32.484248    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"]
Sep 10 21:07:32.486529 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:07:32.486499    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fdc9cd_5090_43b6_b765_f65f80219f4f.slice/crio-73e094b8d64371cb13c19ec6d6817272cc9a41b71de128408b455ef7d4589ff2 WatchSource:0}: Error finding container 73e094b8d64371cb13c19ec6d6817272cc9a41b71de128408b455ef7d4589ff2: Status 404 returned error can't find the container with id 73e094b8d64371cb13c19ec6d6817272cc9a41b71de128408b455ef7d4589ff2
Sep 10 21:07:32.928338 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:32.928313    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526" event={"ID":"46fdc9cd-5090-43b6-b765-f65f80219f4f","Type":"ContainerStarted","Data":"73e094b8d64371cb13c19ec6d6817272cc9a41b71de128408b455ef7d4589ff2"}
Sep 10 21:07:33.936228 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:33.936195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526" event={"ID":"46fdc9cd-5090-43b6-b765-f65f80219f4f","Type":"ContainerStarted","Data":"e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96"}
Sep 10 21:07:33.936623 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:33.936319    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:33.960301 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:33.960256    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526" podStartSLOduration=2.547025604 podStartE2EDuration="2.960241653s" totalImagesPullingTime="413.216049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:33.959000329 +0000 UTC m=+1889.870037613" watchObservedRunningTime="2026-09-10 21:07:33.960241653 +0000 UTC m=+1889.871278978"
Sep 10 21:07:39.947641 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:39.947613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:07:40.202294 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:07:40.202218    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:08:02.002995 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.001718    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"]
Sep 10 21:08:02.002995 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.001990    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526" podUID="46fdc9cd-5090-43b6-b765-f65f80219f4f" containerName="maas-api" containerID="cri-o://e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96" gracePeriod=30
Sep 10 21:08:02.250175 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.250150    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:08:02.312353 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.312284    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fdc9cd-5090-43b6-b765-f65f80219f4f-openshift-service-ca\") pod \"46fdc9cd-5090-43b6-b765-f65f80219f4f\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") "
Sep 10 21:08:02.312353 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.312345    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls\") pod \"46fdc9cd-5090-43b6-b765-f65f80219f4f\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") "
Sep 10 21:08:02.312509 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.312365    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4bhw\" (UniqueName: \"kubernetes.io/projected/46fdc9cd-5090-43b6-b765-f65f80219f4f-kube-api-access-m4bhw\") pod \"46fdc9cd-5090-43b6-b765-f65f80219f4f\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") "
Sep 10 21:08:02.312509 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.312449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-metrics-certs\") pod \"46fdc9cd-5090-43b6-b765-f65f80219f4f\" (UID: \"46fdc9cd-5090-43b6-b765-f65f80219f4f\") "
Sep 10 21:08:02.312681 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.312656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fdc9cd-5090-43b6-b765-f65f80219f4f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46fdc9cd-5090-43b6-b765-f65f80219f4f" (UID: "46fdc9cd-5090-43b6-b765-f65f80219f4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:08:02.314515 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.314487    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fdc9cd-5090-43b6-b765-f65f80219f4f-kube-api-access-m4bhw" (OuterVolumeSpecName: "kube-api-access-m4bhw") pod "46fdc9cd-5090-43b6-b765-f65f80219f4f" (UID: "46fdc9cd-5090-43b6-b765-f65f80219f4f"). InnerVolumeSpecName "kube-api-access-m4bhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:08:02.314515 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.314502    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46fdc9cd-5090-43b6-b765-f65f80219f4f" (UID: "46fdc9cd-5090-43b6-b765-f65f80219f4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:08:02.314705 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.314566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "46fdc9cd-5090-43b6-b765-f65f80219f4f" (UID: "46fdc9cd-5090-43b6-b765-f65f80219f4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:08:02.413166 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.413137    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:02.413166 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.413164    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bhw\" (UniqueName: \"kubernetes.io/projected/46fdc9cd-5090-43b6-b765-f65f80219f4f-kube-api-access-m4bhw\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:02.413166 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.413173    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fdc9cd-5090-43b6-b765-f65f80219f4f-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:02.413421 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:02.413182    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fdc9cd-5090-43b6-b765-f65f80219f4f-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:03.119341 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.119308    2565 generic.go:356] "Generic (PLEG): container finished" podID="46fdc9cd-5090-43b6-b765-f65f80219f4f" containerID="e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96" exitCode=0
Sep 10 21:08:03.119740 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.119349    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"
Sep 10 21:08:03.119740 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.119380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526" event={"ID":"46fdc9cd-5090-43b6-b765-f65f80219f4f","Type":"ContainerDied","Data":"e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96"}
Sep 10 21:08:03.119740 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.119421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526" event={"ID":"46fdc9cd-5090-43b6-b765-f65f80219f4f","Type":"ContainerDied","Data":"73e094b8d64371cb13c19ec6d6817272cc9a41b71de128408b455ef7d4589ff2"}
Sep 10 21:08:03.119740 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.119441    2565 scope.go:117] "RemoveContainer" containerID="e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96"
Sep 10 21:08:03.128777 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.128761    2565 scope.go:117] "RemoveContainer" containerID="e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96"
Sep 10 21:08:03.129031 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:08:03.129015    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96\": container with ID starting with e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96 not found: ID does not exist" containerID="e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96"
Sep 10 21:08:03.129079 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.129039    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96"} err="failed to get container status \"e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96\": rpc error: code = NotFound desc = could not find container \"e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96\": container with ID starting with e129dd7af1c851234402108ed2da2790fa904488c2e08c071baf55aff31c0b96 not found: ID does not exist"
Sep 10 21:08:03.139669 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.139648    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"]
Sep 10 21:08:03.144183 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:03.144161    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb2a90-6fb46f674b-7n526"]
Sep 10 21:08:04.591618 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:04.591577    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46fdc9cd-5090-43b6-b765-f65f80219f4f" path="/var/lib/kubelet/pods/46fdc9cd-5090-43b6-b765-f65f80219f4f/volumes"
Sep 10 21:08:06.204240 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:06.204203    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:08:25.905368 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:25.905334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:08:28.205834 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.205802    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"]
Sep 10 21:08:28.206261 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.206016    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q" podUID="8bd36734-a95a-4ac4-9a09-aec24cccb0b7" containerName="maas-api" containerID="cri-o://56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d" gracePeriod=30
Sep 10 21:08:28.460441 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.460373    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:08:28.641870 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.641835    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-openshift-service-ca\") pod \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") "
Sep 10 21:08:28.642017 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.641892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls\") pod \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") "
Sep 10 21:08:28.642017 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.641916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-metrics-certs\") pod \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") "
Sep 10 21:08:28.642017 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.641955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khlhp\" (UniqueName: \"kubernetes.io/projected/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-kube-api-access-khlhp\") pod \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\" (UID: \"8bd36734-a95a-4ac4-9a09-aec24cccb0b7\") "
Sep 10 21:08:28.642266 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.642235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bd36734-a95a-4ac4-9a09-aec24cccb0b7" (UID: "8bd36734-a95a-4ac4-9a09-aec24cccb0b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:08:28.644127 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.644100    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bd36734-a95a-4ac4-9a09-aec24cccb0b7" (UID: "8bd36734-a95a-4ac4-9a09-aec24cccb0b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:08:28.644257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.644187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8bd36734-a95a-4ac4-9a09-aec24cccb0b7" (UID: "8bd36734-a95a-4ac4-9a09-aec24cccb0b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:08:28.644257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.644187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-kube-api-access-khlhp" (OuterVolumeSpecName: "kube-api-access-khlhp") pod "8bd36734-a95a-4ac4-9a09-aec24cccb0b7" (UID: "8bd36734-a95a-4ac4-9a09-aec24cccb0b7"). InnerVolumeSpecName "kube-api-access-khlhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:08:28.744092 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.743901    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:28.744092 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.743933    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:28.744092 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.743948    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:28.744092 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:28.743963    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khlhp\" (UniqueName: \"kubernetes.io/projected/8bd36734-a95a-4ac4-9a09-aec24cccb0b7-kube-api-access-khlhp\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:08:29.281221 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.281189    2565 generic.go:356] "Generic (PLEG): container finished" podID="8bd36734-a95a-4ac4-9a09-aec24cccb0b7" containerID="56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d" exitCode=0
Sep 10 21:08:29.281634 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.281243    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"
Sep 10 21:08:29.281634 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.281274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q" event={"ID":"8bd36734-a95a-4ac4-9a09-aec24cccb0b7","Type":"ContainerDied","Data":"56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d"}
Sep 10 21:08:29.281634 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.281349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q" event={"ID":"8bd36734-a95a-4ac4-9a09-aec24cccb0b7","Type":"ContainerDied","Data":"04e05f61f52f86472b41ca1546f9d2b4ee766cf7e44215b43659f5257989e007"}
Sep 10 21:08:29.281634 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.281372    2565 scope.go:117] "RemoveContainer" containerID="56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d"
Sep 10 21:08:29.291395 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.291381    2565 scope.go:117] "RemoveContainer" containerID="56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d"
Sep 10 21:08:29.291657 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:08:29.291641    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d\": container with ID starting with 56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d not found: ID does not exist" containerID="56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d"
Sep 10 21:08:29.291755 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.291664    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d"} err="failed to get container status \"56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d\": rpc error: code = NotFound desc = could not find container \"56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d\": container with ID starting with 56afa045267c162a507d8e0cda32fa9d52da52f9817e9f2892d431a18844f32d not found: ID does not exist"
Sep 10 21:08:29.303465 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.303442    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"]
Sep 10 21:08:29.309246 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:29.309227    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73da3-798fdcfc44-v4v8q"]
Sep 10 21:08:30.591345 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:30.591309    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd36734-a95a-4ac4-9a09-aec24cccb0b7" path="/var/lib/kubelet/pods/8bd36734-a95a-4ac4-9a09-aec24cccb0b7/volumes"
Sep 10 21:08:38.899878 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:08:38.899843    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:09:04.993085 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:04.992981    2565 scope.go:117] "RemoveContainer" containerID="7671082f9d8b36a5fd4496d10e0b2238c92dd15d835f45626186f7c6f9a20883"
Sep 10 21:09:28.920249 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:28.920215    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"]
Sep 10 21:09:28.920788 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:28.920422    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4" podUID="07f5c866-ffc1-4731-b546-0222f3994056" containerName="maas-api" containerID="cri-o://dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6" gracePeriod=30
Sep 10 21:09:29.160671 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.160646    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:09:29.266659 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.266585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-maas-api-tls\") pod \"07f5c866-ffc1-4731-b546-0222f3994056\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") "
Sep 10 21:09:29.266659 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.266618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-metrics-certs\") pod \"07f5c866-ffc1-4731-b546-0222f3994056\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") "
Sep 10 21:09:29.266888 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.266681    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swvcl\" (UniqueName: \"kubernetes.io/projected/07f5c866-ffc1-4731-b546-0222f3994056-kube-api-access-swvcl\") pod \"07f5c866-ffc1-4731-b546-0222f3994056\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") "
Sep 10 21:09:29.266888 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.266736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f5c866-ffc1-4731-b546-0222f3994056-openshift-service-ca\") pod \"07f5c866-ffc1-4731-b546-0222f3994056\" (UID: \"07f5c866-ffc1-4731-b546-0222f3994056\") "
Sep 10 21:09:29.267225 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.267195    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f5c866-ffc1-4731-b546-0222f3994056-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07f5c866-ffc1-4731-b546-0222f3994056" (UID: "07f5c866-ffc1-4731-b546-0222f3994056"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:29.269336 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.269310    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07f5c866-ffc1-4731-b546-0222f3994056" (UID: "07f5c866-ffc1-4731-b546-0222f3994056"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:09:29.269444 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.269339    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07f5c866-ffc1-4731-b546-0222f3994056" (UID: "07f5c866-ffc1-4731-b546-0222f3994056"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:09:29.269444 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.269355    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f5c866-ffc1-4731-b546-0222f3994056-kube-api-access-swvcl" (OuterVolumeSpecName: "kube-api-access-swvcl") pod "07f5c866-ffc1-4731-b546-0222f3994056" (UID: "07f5c866-ffc1-4731-b546-0222f3994056"). InnerVolumeSpecName "kube-api-access-swvcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:29.367563 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.367530    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f5c866-ffc1-4731-b546-0222f3994056-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.367563 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.367559    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.367563 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.367568    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f5c866-ffc1-4731-b546-0222f3994056-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.367807 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.367578    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvcl\" (UniqueName: \"kubernetes.io/projected/07f5c866-ffc1-4731-b546-0222f3994056-kube-api-access-swvcl\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.614252 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.614215    2565 generic.go:356] "Generic (PLEG): container finished" podID="07f5c866-ffc1-4731-b546-0222f3994056" containerID="dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6" exitCode=0
Sep 10 21:09:29.614252 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.614250    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"
Sep 10 21:09:29.614456 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.614304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4" event={"ID":"07f5c866-ffc1-4731-b546-0222f3994056","Type":"ContainerDied","Data":"dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6"}
Sep 10 21:09:29.614456 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.614354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4" event={"ID":"07f5c866-ffc1-4731-b546-0222f3994056","Type":"ContainerDied","Data":"62a6c3abbf8f3850cb6cace4e9fa834c1098707baf367a41c918ba24cafa8bf0"}
Sep 10 21:09:29.614456 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.614377    2565 scope.go:117] "RemoveContainer" containerID="dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6"
Sep 10 21:09:29.625944 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.625912    2565 scope.go:117] "RemoveContainer" containerID="dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6"
Sep 10 21:09:29.626380 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:09:29.626348    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6\": container with ID starting with dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6 not found: ID does not exist" containerID="dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6"
Sep 10 21:09:29.626503 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.626390    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6"} err="failed to get container status \"dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6\": rpc error: code = NotFound desc = could not find container \"dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6\": container with ID starting with dfe75d8248c2367edbc5ba1c91e309bafc9217fb2f3e38dea93b6dbfed7ae6a6 not found: ID does not exist"
Sep 10 21:09:29.652608 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.652578    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"]
Sep 10 21:09:29.655536 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:29.655515    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18b9ac-784678c989-qmfq4"]
Sep 10 21:09:30.591352 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:30.591317    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f5c866-ffc1-4731-b546-0222f3994056" path="/var/lib/kubelet/pods/07f5c866-ffc1-4731-b546-0222f3994056/volumes"
Sep 10 21:09:32.392286 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:32.392250    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:09:46.815464 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:46.815431    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"]
Sep 10 21:09:46.815910 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:46.815727    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw" podUID="2c228527-cca0-4b08-a949-a138f8027dc1" containerName="maas-api" containerID="cri-o://66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc" gracePeriod=30
Sep 10 21:09:47.072968 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.072908    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 21:09:47.120677 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.120645    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c228527-cca0-4b08-a949-a138f8027dc1-openshift-service-ca\") pod \"2c228527-cca0-4b08-a949-a138f8027dc1\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") "
Sep 10 21:09:47.120848 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.120755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-metrics-certs\") pod \"2c228527-cca0-4b08-a949-a138f8027dc1\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") "
Sep 10 21:09:47.120930 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.120911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhvch\" (UniqueName: \"kubernetes.io/projected/2c228527-cca0-4b08-a949-a138f8027dc1-kube-api-access-qhvch\") pod \"2c228527-cca0-4b08-a949-a138f8027dc1\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") "
Sep 10 21:09:47.121019 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.121000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c228527-cca0-4b08-a949-a138f8027dc1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c228527-cca0-4b08-a949-a138f8027dc1" (UID: "2c228527-cca0-4b08-a949-a138f8027dc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:47.121086 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.121045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls\") pod \"2c228527-cca0-4b08-a949-a138f8027dc1\" (UID: \"2c228527-cca0-4b08-a949-a138f8027dc1\") "
Sep 10 21:09:47.121391 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.121370    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c228527-cca0-4b08-a949-a138f8027dc1-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:47.123019 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.122992    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c228527-cca0-4b08-a949-a138f8027dc1-kube-api-access-qhvch" (OuterVolumeSpecName: "kube-api-access-qhvch") pod "2c228527-cca0-4b08-a949-a138f8027dc1" (UID: "2c228527-cca0-4b08-a949-a138f8027dc1"). InnerVolumeSpecName "kube-api-access-qhvch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:47.123275 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.123251    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2c228527-cca0-4b08-a949-a138f8027dc1" (UID: "2c228527-cca0-4b08-a949-a138f8027dc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:09:47.123424 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.123403    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c228527-cca0-4b08-a949-a138f8027dc1" (UID: "2c228527-cca0-4b08-a949-a138f8027dc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:09:47.222285 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.222264    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:47.222285 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.222285    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c228527-cca0-4b08-a949-a138f8027dc1-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:47.222427 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.222293    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhvch\" (UniqueName: \"kubernetes.io/projected/2c228527-cca0-4b08-a949-a138f8027dc1-kube-api-access-qhvch\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:09:47.712368 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.712335    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c228527-cca0-4b08-a949-a138f8027dc1" containerID="66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc" exitCode=0
Sep 10 21:09:47.712531 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.712378    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"
Sep 10 21:09:47.712531 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.712422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw" event={"ID":"2c228527-cca0-4b08-a949-a138f8027dc1","Type":"ContainerDied","Data":"66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc"}
Sep 10 21:09:47.712531 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.712480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw" event={"ID":"2c228527-cca0-4b08-a949-a138f8027dc1","Type":"ContainerDied","Data":"a98c85681aac5777cbb69e2c25b1f70578ff76804be4ce133dfe25b482ee4d71"}
Sep 10 21:09:47.712531 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.712510    2565 scope.go:117] "RemoveContainer" containerID="66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc"
Sep 10 21:09:47.724109 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.724090    2565 scope.go:117] "RemoveContainer" containerID="66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc"
Sep 10 21:09:47.724342 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:09:47.724326    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc\": container with ID starting with 66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc not found: ID does not exist" containerID="66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc"
Sep 10 21:09:47.724384 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.724352    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc"} err="failed to get container status \"66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc\": rpc error: code = NotFound desc = could not find container \"66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc\": container with ID starting with 66ea09f7f984ad29b755f5886f3311d3a5f69a955357a968dfc00768079b32fc not found: ID does not exist"
Sep 10 21:09:47.739534 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.739510    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"]
Sep 10 21:09:47.744067 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:47.744046    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3fc2b8-7969cf955f-p2mcw"]
Sep 10 21:09:48.591401 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:48.591367    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c228527-cca0-4b08-a949-a138f8027dc1" path="/var/lib/kubelet/pods/2c228527-cca0-4b08-a949-a138f8027dc1/volumes"
Sep 10 21:09:49.700805 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:09:49.700770    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:10:05.076982 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.076942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"]
Sep 10 21:10:05.077503 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.077146    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" podUID="a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" containerName="maas-api" containerID="cri-o://c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed" gracePeriod=30
Sep 10 21:10:05.324542 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.322780    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 21:10:05.479404 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.479369    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-openshift-service-ca\") pod \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") "
Sep 10 21:10:05.479404 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.479408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls\") pod \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") "
Sep 10 21:10:05.479662 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.479472    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-metrics-certs\") pod \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") "
Sep 10 21:10:05.479662 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.479488    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg6d7\" (UniqueName: \"kubernetes.io/projected/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-kube-api-access-zg6d7\") pod \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\" (UID: \"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7\") "
Sep 10 21:10:05.479893 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.479864    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" (UID: "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:05.481622 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.481586    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" (UID: "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:10:05.481622 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.481604    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" (UID: "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:10:05.481779 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.481640    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-kube-api-access-zg6d7" (OuterVolumeSpecName: "kube-api-access-zg6d7") pod "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" (UID: "a85ae8e6-bba2-4deb-83ed-3276ff0d09a7"). InnerVolumeSpecName "kube-api-access-zg6d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:05.580392 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.580320    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:10:05.580392 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.580346    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:10:05.580392 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.580355    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:10:05.580392 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.580364    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6d7\" (UniqueName: \"kubernetes.io/projected/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7-kube-api-access-zg6d7\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:10:05.807370 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.807335    2565 generic.go:356] "Generic (PLEG): container finished" podID="a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" containerID="c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed" exitCode=0
Sep 10 21:10:05.807556 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.807380    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"
Sep 10 21:10:05.807556 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.807412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" event={"ID":"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7","Type":"ContainerDied","Data":"c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed"}
Sep 10 21:10:05.807556 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.807466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp" event={"ID":"a85ae8e6-bba2-4deb-83ed-3276ff0d09a7","Type":"ContainerDied","Data":"5129a7e0eb5cd1278cf3b44a19b05ac95ac15f88d120760c156ded48bcdda3bc"}
Sep 10 21:10:05.807556 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.807492    2565 scope.go:117] "RemoveContainer" containerID="c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed"
Sep 10 21:10:05.818223 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.818203    2565 scope.go:117] "RemoveContainer" containerID="c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed"
Sep 10 21:10:05.818459 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:10:05.818441    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed\": container with ID starting with c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed not found: ID does not exist" containerID="c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed"
Sep 10 21:10:05.818540 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.818466    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed"} err="failed to get container status \"c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed\": rpc error: code = NotFound desc = could not find container \"c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed\": container with ID starting with c7e660129a1038c6a1169838c1eaecf0abfc6dcf55bafe87fe3e9720d45209ed not found: ID does not exist"
Sep 10 21:10:05.830844 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.830791    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"]
Sep 10 21:10:05.834528 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:05.834506    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4fe4f-d796c9f49-8mlrp"]
Sep 10 21:10:06.591259 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:06.591222    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" path="/var/lib/kubelet/pods/a85ae8e6-bba2-4deb-83ed-3276ff0d09a7/volumes"
Sep 10 21:10:07.697164 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:07.697132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:10:16.394663 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:10:16.394627    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:11:04.728501 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:11:04.728475    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:11:04.732540 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:11:04.732520    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:11:04.733828 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:11:04.733811    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:11:04.737804 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:11:04.737787    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:11:35.994272 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:11:35.994242    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:12:01.195355 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:01.195276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:12:08.996716 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:08.996668    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:12:17.332363 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:17.332326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:12:34.900046 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:34.900011    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:12:40.373555 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:40.373522    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"]
Sep 10 21:12:40.374093 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:40.373748    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw" podUID="b6e17378-c89c-42ec-925f-b79c9d34eec3" containerName="manager" containerID="cri-o://c4fd859d33846cc42a7192e3d3318a9ca7f90b7066a6cc824a0a03780c6db9fa" gracePeriod=10
Sep 10 21:12:42.653068 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:42.653041    2565 generic.go:356] "Generic (PLEG): container finished" podID="b6e17378-c89c-42ec-925f-b79c9d34eec3" containerID="c4fd859d33846cc42a7192e3d3318a9ca7f90b7066a6cc824a0a03780c6db9fa" exitCode=0
Sep 10 21:12:42.653367 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:42.653114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw" event={"ID":"b6e17378-c89c-42ec-925f-b79c9d34eec3","Type":"ContainerDied","Data":"c4fd859d33846cc42a7192e3d3318a9ca7f90b7066a6cc824a0a03780c6db9fa"}
Sep 10 21:12:42.755221 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:42.755185    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw" podUID="b6e17378-c89c-42ec-925f-b79c9d34eec3" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": dial tcp 10.133.0.44:8081: connect: connection refused"
Sep 10 21:12:43.225825 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.225803    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 21:12:43.304400 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.304369    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e17378-c89c-42ec-925f-b79c9d34eec3-extensions-socket-volume\") pod \"b6e17378-c89c-42ec-925f-b79c9d34eec3\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") "
Sep 10 21:12:43.304578 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.304520    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hznxk\" (UniqueName: \"kubernetes.io/projected/b6e17378-c89c-42ec-925f-b79c9d34eec3-kube-api-access-hznxk\") pod \"b6e17378-c89c-42ec-925f-b79c9d34eec3\" (UID: \"b6e17378-c89c-42ec-925f-b79c9d34eec3\") "
Sep 10 21:12:43.304840 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.304814    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e17378-c89c-42ec-925f-b79c9d34eec3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b6e17378-c89c-42ec-925f-b79c9d34eec3" (UID: "b6e17378-c89c-42ec-925f-b79c9d34eec3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:43.306459 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.306436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e17378-c89c-42ec-925f-b79c9d34eec3-kube-api-access-hznxk" (OuterVolumeSpecName: "kube-api-access-hznxk") pod "b6e17378-c89c-42ec-925f-b79c9d34eec3" (UID: "b6e17378-c89c-42ec-925f-b79c9d34eec3"). InnerVolumeSpecName "kube-api-access-hznxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:43.405797 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.405768    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e17378-c89c-42ec-925f-b79c9d34eec3-extensions-socket-volume\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:12:43.405797 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.405793    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznxk\" (UniqueName: \"kubernetes.io/projected/b6e17378-c89c-42ec-925f-b79c9d34eec3-kube-api-access-hznxk\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:12:43.660114 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.660040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw" event={"ID":"b6e17378-c89c-42ec-925f-b79c9d34eec3","Type":"ContainerDied","Data":"5884e7f9d9501761c77876e65cffdd814b96a861da266671d94ea02a3e5b2df2"}
Sep 10 21:12:43.660114 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.660062    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"
Sep 10 21:12:43.660114 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.660098    2565 scope.go:117] "RemoveContainer" containerID="c4fd859d33846cc42a7192e3d3318a9ca7f90b7066a6cc824a0a03780c6db9fa"
Sep 10 21:12:43.682634 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.682611    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"]
Sep 10 21:12:43.686888 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:43.686867    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bsdzw"]
Sep 10 21:12:44.591667 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:12:44.591636    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6e17378-c89c-42ec-925f-b79c9d34eec3" path="/var/lib/kubelet/pods/b6e17378-c89c-42ec-925f-b79c9d34eec3/volumes"
Sep 10 21:13:41.074413 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074338    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"]
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074759    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c228527-cca0-4b08-a949-a138f8027dc1" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074772    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c228527-cca0-4b08-a949-a138f8027dc1" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46fdc9cd-5090-43b6-b765-f65f80219f4f" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074791    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46fdc9cd-5090-43b6-b765-f65f80219f4f" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074801    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e17378-c89c-42ec-925f-b79c9d34eec3" containerName="manager"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074808    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e17378-c89c-42ec-925f-b79c9d34eec3" containerName="manager"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074815    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074821    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074826    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd36734-a95a-4ac4-9a09-aec24cccb0b7" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074831    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd36734-a95a-4ac4-9a09-aec24cccb0b7" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074841    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f5c866-ffc1-4731-b546-0222f3994056" containerName="maas-api"
Sep 10 21:13:41.074886 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074846    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f5c866-ffc1-4731-b546-0222f3994056" containerName="maas-api"
Sep 10 21:13:41.075257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074906    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd36734-a95a-4ac4-9a09-aec24cccb0b7" containerName="maas-api"
Sep 10 21:13:41.075257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074913    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46fdc9cd-5090-43b6-b765-f65f80219f4f" containerName="maas-api"
Sep 10 21:13:41.075257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074920    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85ae8e6-bba2-4deb-83ed-3276ff0d09a7" containerName="maas-api"
Sep 10 21:13:41.075257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074929    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e17378-c89c-42ec-925f-b79c9d34eec3" containerName="manager"
Sep 10 21:13:41.075257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074936    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c228527-cca0-4b08-a949-a138f8027dc1" containerName="maas-api"
Sep 10 21:13:41.075257 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.074940    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f5c866-ffc1-4731-b546-0222f3994056" containerName="maas-api"
Sep 10 21:13:41.080763 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.079712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.083410 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.083387    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jzjqd\""
Sep 10 21:13:41.089144 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.089115    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"]
Sep 10 21:13:41.184929 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.184894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34efc22e-e47d-4106-9501-76510906ac9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n6qq5\" (UID: \"34efc22e-e47d-4106-9501-76510906ac9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.185089 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.184961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmfv\" (UniqueName: \"kubernetes.io/projected/34efc22e-e47d-4106-9501-76510906ac9a-kube-api-access-lqmfv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n6qq5\" (UID: \"34efc22e-e47d-4106-9501-76510906ac9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.286226 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.286192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34efc22e-e47d-4106-9501-76510906ac9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n6qq5\" (UID: \"34efc22e-e47d-4106-9501-76510906ac9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.286409 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.286253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmfv\" (UniqueName: \"kubernetes.io/projected/34efc22e-e47d-4106-9501-76510906ac9a-kube-api-access-lqmfv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n6qq5\" (UID: \"34efc22e-e47d-4106-9501-76510906ac9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.286556 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.286538    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34efc22e-e47d-4106-9501-76510906ac9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n6qq5\" (UID: \"34efc22e-e47d-4106-9501-76510906ac9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.295506 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.295479    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmfv\" (UniqueName: \"kubernetes.io/projected/34efc22e-e47d-4106-9501-76510906ac9a-kube-api-access-lqmfv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n6qq5\" (UID: \"34efc22e-e47d-4106-9501-76510906ac9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.392938 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.392911    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.665771 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.665746    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"]
Sep 10 21:13:41.667607 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:13:41.667582    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34efc22e_e47d_4106_9501_76510906ac9a.slice/crio-68e992311184eee7e24c801a153f53df86a4a2e217db8fd991adac7b3c33c154 WatchSource:0}: Error finding container 68e992311184eee7e24c801a153f53df86a4a2e217db8fd991adac7b3c33c154: Status 404 returned error can't find the container with id 68e992311184eee7e24c801a153f53df86a4a2e217db8fd991adac7b3c33c154
Sep 10 21:13:41.965110 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.965027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5" event={"ID":"34efc22e-e47d-4106-9501-76510906ac9a","Type":"ContainerStarted","Data":"1f81c5a2c90493ee243509a870ace569fbfba049d2a2439f5e68bf8538177af7"}
Sep 10 21:13:41.965110 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.965078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5" event={"ID":"34efc22e-e47d-4106-9501-76510906ac9a","Type":"ContainerStarted","Data":"68e992311184eee7e24c801a153f53df86a4a2e217db8fd991adac7b3c33c154"}
Sep 10 21:13:41.965351 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.965182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:13:41.987463 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:41.987422    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5" podStartSLOduration=0.987409117 podStartE2EDuration="987.409117ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:41.987021642 +0000 UTC m=+2257.898058926" watchObservedRunningTime="2026-09-10 21:13:41.987409117 +0000 UTC m=+2257.898446401"
Sep 10 21:13:52.972557 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:13:52.972520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n6qq5"
Sep 10 21:14:03.103871 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:14:03.103833    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:14:11.101074 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:14:11.101043    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:14:16.993806 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:14:16.993768    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:14:35.892005 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:14:35.891973    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:14:53.495291 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:14:53.495260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:15:00.132982 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:00.132905    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d"]
Sep 10 21:15:00.143054 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:00.143026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d"
Sep 10 21:15:00.146457 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:00.146396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d"]
Sep 10 21:15:00.146962 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:00.146934    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:15:00.155510 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:00.155492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d"
Sep 10 21:15:00.426790 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:00.426763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d"]
Sep 10 21:15:00.429536 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:15:00.429506    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73822d5b_e577_42b8_b993_db6146cfd818.slice/crio-c033a032afbccae6d65111b531d16311bcf4d5a5d9d4b77e93fda596978ba2e6 WatchSource:0}: Error finding container c033a032afbccae6d65111b531d16311bcf4d5a5d9d4b77e93fda596978ba2e6: Status 404 returned error can't find the container with id c033a032afbccae6d65111b531d16311bcf4d5a5d9d4b77e93fda596978ba2e6
Sep 10 21:15:01.396939 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:01.396906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d" event={"ID":"73822d5b-e577-42b8-b993-db6146cfd818","Type":"ContainerStarted","Data":"37436ad1b61accc7652f6853285188238e50b109408212a81f9b7107a81f8099"}
Sep 10 21:15:01.397310 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:01.396960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d" event={"ID":"73822d5b-e577-42b8-b993-db6146cfd818","Type":"ContainerStarted","Data":"c033a032afbccae6d65111b531d16311bcf4d5a5d9d4b77e93fda596978ba2e6"}
Sep 10 21:15:01.422641 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:01.422596    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d" podStartSLOduration=1.422581555 podStartE2EDuration="1.422581555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:01.420894299 +0000 UTC m=+2337.331931584" watchObservedRunningTime="2026-09-10 21:15:01.422581555 +0000 UTC m=+2337.333618880"
Sep 10 21:15:02.403413 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:02.403379    2565 generic.go:356] "Generic (PLEG): container finished" podID="73822d5b-e577-42b8-b993-db6146cfd818" containerID="37436ad1b61accc7652f6853285188238e50b109408212a81f9b7107a81f8099" exitCode=0
Sep 10 21:15:02.403826 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:02.403469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d" event={"ID":"73822d5b-e577-42b8-b993-db6146cfd818","Type":"ContainerDied","Data":"37436ad1b61accc7652f6853285188238e50b109408212a81f9b7107a81f8099"}
Sep 10 21:15:03.534846 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:03.534821    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d"
Sep 10 21:15:04.415431 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:04.415403    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d"
Sep 10 21:15:04.415431 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:04.415413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-whb6d" event={"ID":"73822d5b-e577-42b8-b993-db6146cfd818","Type":"ContainerDied","Data":"c033a032afbccae6d65111b531d16311bcf4d5a5d9d4b77e93fda596978ba2e6"}
Sep 10 21:15:04.415643 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:04.415458    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c033a032afbccae6d65111b531d16311bcf4d5a5d9d4b77e93fda596978ba2e6"
Sep 10 21:15:18.718554 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:18.718514    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:15:27.902209 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:27.902173    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:15:34.900194 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:34.900155    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:15:41.526069 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.526038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:15:41.902555 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.902519    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"]
Sep 10 21:15:41.902960 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.902947    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73822d5b-e577-42b8-b993-db6146cfd818" containerName="cleanup"
Sep 10 21:15:41.902960 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.902961    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73822d5b-e577-42b8-b993-db6146cfd818" containerName="cleanup"
Sep 10 21:15:41.903060 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.903034    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73822d5b-e577-42b8-b993-db6146cfd818" containerName="cleanup"
Sep 10 21:15:41.906137 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.906119    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:41.909755 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.909733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqzh2\""
Sep 10 21:15:41.909871 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.909738    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:15:41.910990 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.910972    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:15:41.910990 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.910981    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:15:41.911142 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.910980    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 21:15:41.918219 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.918196    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"]
Sep 10 21:15:41.920513 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.920491    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"]
Sep 10 21:15:41.924538 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.924520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:41.934988 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:41.934969    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"]
Sep 10 21:15:42.030171 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-maas-api-tls\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.030171 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xl9\" (UniqueName: \"kubernetes.io/projected/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-kube-api-access-f5xl9\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.030417 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-maas-api-tls\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.030417 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-metrics-certs\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.030417 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-openshift-service-ca\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.030417 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-metrics-certs\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.030417 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54260376-dc35-4adf-8564-2cc2cad46d2f-openshift-service-ca\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.030596 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.030438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65skv\" (UniqueName: \"kubernetes.io/projected/54260376-dc35-4adf-8564-2cc2cad46d2f-kube-api-access-65skv\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.131526 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65skv\" (UniqueName: \"kubernetes.io/projected/54260376-dc35-4adf-8564-2cc2cad46d2f-kube-api-access-65skv\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.131671 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-maas-api-tls\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.131671 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xl9\" (UniqueName: \"kubernetes.io/projected/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-kube-api-access-f5xl9\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.131671 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-maas-api-tls\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.131671 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-metrics-certs\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.131889 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-openshift-service-ca\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.131889 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-metrics-certs\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.131889 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.131760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54260376-dc35-4adf-8564-2cc2cad46d2f-openshift-service-ca\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.132493 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.132465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54260376-dc35-4adf-8564-2cc2cad46d2f-openshift-service-ca\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.132631 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.132514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-openshift-service-ca\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.134426 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.134399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-metrics-certs\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.134532 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.134425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-maas-api-tls\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.134633 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.134613    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-maas-api-tls\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.134889 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.134870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-metrics-certs\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.141535 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.141506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xl9\" (UniqueName: \"kubernetes.io/projected/8b0deab3-7ce0-40de-8dcf-50eafe506e4b-kube-api-access-f5xl9\") pod \"maas-api-7c74b8f674-6j7gs\" (UID: \"8b0deab3-7ce0-40de-8dcf-50eafe506e4b\") " pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.141959 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.141941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65skv\" (UniqueName: \"kubernetes.io/projected/54260376-dc35-4adf-8564-2cc2cad46d2f-kube-api-access-65skv\") pod \"maas-api-5999f65fcd-8q46s\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") " pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.217898 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.217825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:42.236148 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.236125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:42.498942 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.498911    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"]
Sep 10 21:15:42.500369 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:15:42.500341    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54260376_dc35_4adf_8564_2cc2cad46d2f.slice/crio-3aa88e6c0b84e1c01cf87424c19c020ed41b030e3e1c986ab4c389f11c24fe50 WatchSource:0}: Error finding container 3aa88e6c0b84e1c01cf87424c19c020ed41b030e3e1c986ab4c389f11c24fe50: Status 404 returned error can't find the container with id 3aa88e6c0b84e1c01cf87424c19c020ed41b030e3e1c986ab4c389f11c24fe50
Sep 10 21:15:42.501874 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.501856    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:15:42.519088 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.519067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"]
Sep 10 21:15:42.520090 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:15:42.520071    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0deab3_7ce0_40de_8dcf_50eafe506e4b.slice/crio-c5a62b15c57e42963ebe2d30dc44d94bf3b97556b88399f0cedf49b123ec30c5 WatchSource:0}: Error finding container c5a62b15c57e42963ebe2d30dc44d94bf3b97556b88399f0cedf49b123ec30c5: Status 404 returned error can't find the container with id c5a62b15c57e42963ebe2d30dc44d94bf3b97556b88399f0cedf49b123ec30c5
Sep 10 21:15:42.628471 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.628436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s" event={"ID":"54260376-dc35-4adf-8564-2cc2cad46d2f","Type":"ContainerStarted","Data":"3aa88e6c0b84e1c01cf87424c19c020ed41b030e3e1c986ab4c389f11c24fe50"}
Sep 10 21:15:42.629635 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:42.629611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs" event={"ID":"8b0deab3-7ce0-40de-8dcf-50eafe506e4b","Type":"ContainerStarted","Data":"c5a62b15c57e42963ebe2d30dc44d94bf3b97556b88399f0cedf49b123ec30c5"}
Sep 10 21:15:43.637336 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:43.637289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs" event={"ID":"8b0deab3-7ce0-40de-8dcf-50eafe506e4b","Type":"ContainerStarted","Data":"2d20f7a76bb8750059e8837c3de95a5243dee3481a0a6c15cf99f6b09fc3893c"}
Sep 10 21:15:43.637805 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:43.637435    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:43.638961 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:43.638930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s" event={"ID":"54260376-dc35-4adf-8564-2cc2cad46d2f","Type":"ContainerStarted","Data":"e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1"}
Sep 10 21:15:43.639095 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:43.639067    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:43.657220 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:43.657179    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs" podStartSLOduration=2.169036178 podStartE2EDuration="2.657165973s" totalImagesPullingTime="488.129795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:43.655726472 +0000 UTC m=+2379.566763749" watchObservedRunningTime="2026-09-10 21:15:43.657165973 +0000 UTC m=+2379.568203255"
Sep 10 21:15:43.675763 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:43.675727    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s" podStartSLOduration=2.179975783 podStartE2EDuration="2.675714089s" totalImagesPullingTime="495.738306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:43.673833772 +0000 UTC m=+2379.584871084" watchObservedRunningTime="2026-09-10 21:15:43.675714089 +0000 UTC m=+2379.586751363"
Sep 10 21:15:49.297055 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:49.297020    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:15:49.651659 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:49.651630    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:49.652090 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:49.652070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c74b8f674-6j7gs"
Sep 10 21:15:49.721162 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:49.721133    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"]
Sep 10 21:15:49.721351 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:49.721326    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s" podUID="54260376-dc35-4adf-8564-2cc2cad46d2f" containerName="maas-api" containerID="cri-o://e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1" gracePeriod=30
Sep 10 21:15:49.975468 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:49.975442    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:50.101889 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.101848    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65skv\" (UniqueName: \"kubernetes.io/projected/54260376-dc35-4adf-8564-2cc2cad46d2f-kube-api-access-65skv\") pod \"54260376-dc35-4adf-8564-2cc2cad46d2f\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") "
Sep 10 21:15:50.101889 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.101893    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-maas-api-tls\") pod \"54260376-dc35-4adf-8564-2cc2cad46d2f\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") "
Sep 10 21:15:50.102115 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.101937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-metrics-certs\") pod \"54260376-dc35-4adf-8564-2cc2cad46d2f\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") "
Sep 10 21:15:50.102115 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.102004    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54260376-dc35-4adf-8564-2cc2cad46d2f-openshift-service-ca\") pod \"54260376-dc35-4adf-8564-2cc2cad46d2f\" (UID: \"54260376-dc35-4adf-8564-2cc2cad46d2f\") "
Sep 10 21:15:50.102404 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.102373    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54260376-dc35-4adf-8564-2cc2cad46d2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54260376-dc35-4adf-8564-2cc2cad46d2f" (UID: "54260376-dc35-4adf-8564-2cc2cad46d2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:50.104226 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.104200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "54260376-dc35-4adf-8564-2cc2cad46d2f" (UID: "54260376-dc35-4adf-8564-2cc2cad46d2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:50.104517 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.104486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54260376-dc35-4adf-8564-2cc2cad46d2f-kube-api-access-65skv" (OuterVolumeSpecName: "kube-api-access-65skv") pod "54260376-dc35-4adf-8564-2cc2cad46d2f" (UID: "54260376-dc35-4adf-8564-2cc2cad46d2f"). InnerVolumeSpecName "kube-api-access-65skv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:50.104615 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.104539    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54260376-dc35-4adf-8564-2cc2cad46d2f" (UID: "54260376-dc35-4adf-8564-2cc2cad46d2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:50.203078 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.202970    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65skv\" (UniqueName: \"kubernetes.io/projected/54260376-dc35-4adf-8564-2cc2cad46d2f-kube-api-access-65skv\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:15:50.203078 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.203012    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-maas-api-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:15:50.203078 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.203025    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54260376-dc35-4adf-8564-2cc2cad46d2f-metrics-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:15:50.203078 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.203038    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54260376-dc35-4adf-8564-2cc2cad46d2f-openshift-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Sep 10 21:15:50.679749 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.679715    2565 generic.go:356] "Generic (PLEG): container finished" podID="54260376-dc35-4adf-8564-2cc2cad46d2f" containerID="e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1" exitCode=0
Sep 10 21:15:50.679749 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.679743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"
Sep 10 21:15:50.680175 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.679788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s" event={"ID":"54260376-dc35-4adf-8564-2cc2cad46d2f","Type":"ContainerDied","Data":"e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1"}
Sep 10 21:15:50.680175 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.679833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s" event={"ID":"54260376-dc35-4adf-8564-2cc2cad46d2f","Type":"ContainerDied","Data":"3aa88e6c0b84e1c01cf87424c19c020ed41b030e3e1c986ab4c389f11c24fe50"}
Sep 10 21:15:50.680175 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.679854    2565 scope.go:117] "RemoveContainer" containerID="e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1"
Sep 10 21:15:50.688758 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.688738    2565 scope.go:117] "RemoveContainer" containerID="e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1"
Sep 10 21:15:50.689050 ip-10-0-129-103 kubenswrapper[2565]: E0910 21:15:50.689031    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1\": container with ID starting with e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1 not found: ID does not exist" containerID="e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1"
Sep 10 21:15:50.689103 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.689058    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1"} err="failed to get container status \"e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1\": rpc error: code = NotFound desc = could not find container \"e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1\": container with ID starting with e19cd95ebd8af79fc62a3cc544def80500cffd85d261b6daa7caee15f83e0ae1 not found: ID does not exist"
Sep 10 21:15:50.699602 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.699583    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"]
Sep 10 21:15:50.703703 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:50.703667    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5999f65fcd-8q46s"]
Sep 10 21:15:52.591608 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:52.591577    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54260376-dc35-4adf-8564-2cc2cad46d2f" path="/var/lib/kubelet/pods/54260376-dc35-4adf-8564-2cc2cad46d2f/volumes"
Sep 10 21:15:56.103343 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:15:56.103308    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:16:01.803364 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:01.803327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:16:04.765282 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:04.765254    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:16:04.773171 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:04.773146    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:16:04.774669 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:04.774646    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:16:04.778966 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:04.778949    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwj4x_e98185bd-0c41-43be-bb49-81fe4be59731/ovn-acl-logging/0.log"
Sep 10 21:16:09.303281 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:09.303246    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-275xj"]
Sep 10 21:16:17.270162 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:17.270126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fqgdq_36ff9a55-515f-42ff-98c3-b8ada9d6225e/manager/0.log"
Sep 10 21:16:17.385412 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:17.385376    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xsvmm_9910bb83-a139-4222-afa0-d3bee0b2b885/manager/0.log"
Sep 10 21:16:18.122317 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:18.122286    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-xm2qs_2e2ccf4b-2f6d-46fa-bfe0-b18c41007505/copy-manifests/0.log"
Sep 10 21:16:18.129051 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:18.129025    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-xm2qs_2e2ccf4b-2f6d-46fa-bfe0-b18c41007505/manager/0.log"
Sep 10 21:16:18.462918 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:18.462840    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c74b8f674-6j7gs_8b0deab3-7ce0-40de-8dcf-50eafe506e4b/maas-api/0.log"
Sep 10 21:16:18.794191 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:18.794109    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817915-whb6d_73822d5b-e577-42b8-b993-db6146cfd818/cleanup/0.log"
Sep 10 21:16:19.811357 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:19.811330    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-rdsz7_d6cef623-e76d-46ec-9620-20759029568e/manager/0.log"
Sep 10 21:16:19.919476 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:19.919454    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-mzt4v_22f65693-edbc-47a6-8f45-c4938702a9d1/manager/0.log"
Sep 10 21:16:20.031252 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:20.031223    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bx6fh_378d55bb-2fcb-4da8-ab7c-76906014b4c6/kuadrant-console-plugin/0.log"
Sep 10 21:16:20.147866 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:20.147840    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-n6qq5_34efc22e-e47d-4106-9501-76510906ac9a/manager/0.log"
Sep 10 21:16:20.274147 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:20.274124    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-275xj_380c068d-63eb-496d-a410-56a85b788331/limitador/0.log"
Sep 10 21:16:20.388976 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:20.388952    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kftmv_96295e43-01f0-4f2b-a266-61b61e56285f/manager/0.log"
Sep 10 21:16:21.073921 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:21.073887    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-r45j2_4de93ecd-2faf-4b03-9db2-f6cbb829a253/kube-auth-proxy/0.log"
Sep 10 21:16:21.199762 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:21.199738    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-sspzh_62163f71-e629-4980-81da-76382d8e33ff/istio-proxy/1.log"
Sep 10 21:16:21.444190 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:21.444164    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lq8k2_278ec5f3-c99d-47bb-99d1-0185806fe1e0/payload-processing/0.log"
Sep 10 21:16:22.141454 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:22.141427    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-rvdq7_cdf8d113-d98f-4062-96b2-2f26d2a45f62/main/0.log"
Sep 10 21:16:27.633458 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.633386    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zw7zh/must-gather-qkdqc"]
Sep 10 21:16:27.633853 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.633838    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54260376-dc35-4adf-8564-2cc2cad46d2f" containerName="maas-api"
Sep 10 21:16:27.633853 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.633853    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="54260376-dc35-4adf-8564-2cc2cad46d2f" containerName="maas-api"
Sep 10 21:16:27.633937 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.633925    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="54260376-dc35-4adf-8564-2cc2cad46d2f" containerName="maas-api"
Sep 10 21:16:27.637332 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.637315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:27.640385 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.640365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zw7zh\"/\"default-dockercfg-sck8c\""
Sep 10 21:16:27.640615 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.640596    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zw7zh\"/\"openshift-service-ca.crt\""
Sep 10 21:16:27.640719 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.640696    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zw7zh\"/\"kube-root-ca.crt\""
Sep 10 21:16:27.645884 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.645862    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zw7zh/must-gather-qkdqc"]
Sep 10 21:16:27.724323 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.724298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6944a5ac-4d2c-4b4c-97e7-39d34757b052-must-gather-output\") pod \"must-gather-qkdqc\" (UID: \"6944a5ac-4d2c-4b4c-97e7-39d34757b052\") " pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:27.724436 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.724339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhfb\" (UniqueName: \"kubernetes.io/projected/6944a5ac-4d2c-4b4c-97e7-39d34757b052-kube-api-access-ghhfb\") pod \"must-gather-qkdqc\" (UID: \"6944a5ac-4d2c-4b4c-97e7-39d34757b052\") " pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:27.825810 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.825784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6944a5ac-4d2c-4b4c-97e7-39d34757b052-must-gather-output\") pod \"must-gather-qkdqc\" (UID: \"6944a5ac-4d2c-4b4c-97e7-39d34757b052\") " pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:27.825935 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.825830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhfb\" (UniqueName: \"kubernetes.io/projected/6944a5ac-4d2c-4b4c-97e7-39d34757b052-kube-api-access-ghhfb\") pod \"must-gather-qkdqc\" (UID: \"6944a5ac-4d2c-4b4c-97e7-39d34757b052\") " pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:27.826159 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.826132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6944a5ac-4d2c-4b4c-97e7-39d34757b052-must-gather-output\") pod \"must-gather-qkdqc\" (UID: \"6944a5ac-4d2c-4b4c-97e7-39d34757b052\") " pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:27.835826 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.835797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhfb\" (UniqueName: \"kubernetes.io/projected/6944a5ac-4d2c-4b4c-97e7-39d34757b052-kube-api-access-ghhfb\") pod \"must-gather-qkdqc\" (UID: \"6944a5ac-4d2c-4b4c-97e7-39d34757b052\") " pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:27.947604 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:27.947532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zw7zh/must-gather-qkdqc"
Sep 10 21:16:28.221717 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:28.221672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zw7zh/must-gather-qkdqc"]
Sep 10 21:16:28.222375 ip-10-0-129-103 kubenswrapper[2565]: W0910 21:16:28.222340    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6944a5ac_4d2c_4b4c_97e7_39d34757b052.slice/crio-be943034505844c261b53f4f48f46e8dbd3c4445eaaeb575914c098290a90eff WatchSource:0}: Error finding container be943034505844c261b53f4f48f46e8dbd3c4445eaaeb575914c098290a90eff: Status 404 returned error can't find the container with id be943034505844c261b53f4f48f46e8dbd3c4445eaaeb575914c098290a90eff
Sep 10 21:16:28.894621 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:28.894581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zw7zh/must-gather-qkdqc" event={"ID":"6944a5ac-4d2c-4b4c-97e7-39d34757b052","Type":"ContainerStarted","Data":"be943034505844c261b53f4f48f46e8dbd3c4445eaaeb575914c098290a90eff"}
Sep 10 21:16:29.905934 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:29.905890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zw7zh/must-gather-qkdqc" event={"ID":"6944a5ac-4d2c-4b4c-97e7-39d34757b052","Type":"ContainerStarted","Data":"5058fa1778bf4dc6ad12f50f8fbefc98ca267c12fa2020338a45c16e13dd7f32"}
Sep 10 21:16:29.905934 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:29.905935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zw7zh/must-gather-qkdqc" event={"ID":"6944a5ac-4d2c-4b4c-97e7-39d34757b052","Type":"ContainerStarted","Data":"bcd568442b3238175c2d8b2c02a3201e4f99ff3499e586446a8fbfc7469267b3"}
Sep 10 21:16:29.927128 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:29.927069    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zw7zh/must-gather-qkdqc" podStartSLOduration=2.126889067 podStartE2EDuration="2.927047777s" totalImagesPullingTime="800.15871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:29.924269454 +0000 UTC m=+2425.835306737" watchObservedRunningTime="2026-09-10 21:16:29.927047777 +0000 UTC m=+2425.838085062"
Sep 10 21:16:30.679666 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:30.679632    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-twjd6_159d4776-28fe-4b94-8cc4-85a239b5a5ba/global-pull-secret-syncer/0.log"
Sep 10 21:16:30.828027 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:30.827997    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n7fzz_b7738d02-740f-4e53-9c3d-bcce706af7c7/konnectivity-agent/0.log"
Sep 10 21:16:30.907926 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:30.907894    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-103.ec2.internal_2ab19bc1a57527adb982141bacf53b44/haproxy/0.log"
Sep 10 21:16:35.765463 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:35.765357    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-rdsz7_d6cef623-e76d-46ec-9620-20759029568e/manager/0.log"
Sep 10 21:16:35.800616 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:35.800567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-mzt4v_22f65693-edbc-47a6-8f45-c4938702a9d1/manager/0.log"
Sep 10 21:16:35.830598 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:35.830575    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bx6fh_378d55bb-2fcb-4da8-ab7c-76906014b4c6/kuadrant-console-plugin/0.log"
Sep 10 21:16:35.881974 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:35.881930    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-n6qq5_34efc22e-e47d-4106-9501-76510906ac9a/manager/0.log"
Sep 10 21:16:35.912053 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:35.912027    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-275xj_380c068d-63eb-496d-a410-56a85b788331/limitador/0.log"
Sep 10 21:16:35.992735 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:35.992704    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kftmv_96295e43-01f0-4f2b-a266-61b61e56285f/manager/0.log"
Sep 10 21:16:37.444923 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.444890    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be703d20-4404-44be-a3da-96a045de8bdf/alertmanager/0.log"
Sep 10 21:16:37.473749 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.473716    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be703d20-4404-44be-a3da-96a045de8bdf/config-reloader/0.log"
Sep 10 21:16:37.504759 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.504703    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be703d20-4404-44be-a3da-96a045de8bdf/kube-rbac-proxy-web/0.log"
Sep 10 21:16:37.529028 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.528975    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be703d20-4404-44be-a3da-96a045de8bdf/kube-rbac-proxy/0.log"
Sep 10 21:16:37.555784 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.555759    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be703d20-4404-44be-a3da-96a045de8bdf/kube-rbac-proxy-metric/0.log"
Sep 10 21:16:37.581854 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.581735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be703d20-4404-44be-a3da-96a045de8bdf/prom-label-proxy/0.log"
Sep 10 21:16:37.606537 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.606440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be703d20-4404-44be-a3da-96a045de8bdf/init-config-reloader/0.log"
Sep 10 21:16:37.777362 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:37.777273    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74b7d9f5dc-d28pw_65148aca-66c0-450c-bdce-5dc0fa70e51c/metrics-server/0.log"
Sep 10 21:16:38.043773 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.043745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lkgvr_4c1d3249-e695-467c-af8f-74189a1013a9/node-exporter/0.log"
Sep 10 21:16:38.069623 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.069600    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lkgvr_4c1d3249-e695-467c-af8f-74189a1013a9/kube-rbac-proxy/0.log"
Sep 10 21:16:38.096060 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.096030    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lkgvr_4c1d3249-e695-467c-af8f-74189a1013a9/init-textfile/0.log"
Sep 10 21:16:38.136864 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.136833    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tpgmn_eae173ba-8df5-4919-9647-811bab52d34f/kube-rbac-proxy-main/0.log"
Sep 10 21:16:38.169788 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.169754    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tpgmn_eae173ba-8df5-4919-9647-811bab52d34f/kube-rbac-proxy-self/0.log"
Sep 10 21:16:38.195089 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.195031    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tpgmn_eae173ba-8df5-4919-9647-811bab52d34f/openshift-state-metrics/0.log"
Sep 10 21:16:38.412966 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.412929    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-h5fl7_0e35d50f-376e-4777-a7f0-7aa1d88948b5/prometheus-operator/0.log"
Sep 10 21:16:38.433709 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.433662    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-h5fl7_0e35d50f-376e-4777-a7f0-7aa1d88948b5/kube-rbac-proxy/0.log"
Sep 10 21:16:38.460828 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.460792    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-m6lv6_4b90fe30-6689-4dc5-a19c-b528aa6fdce1/prometheus-operator-admission-webhook/0.log"
Sep 10 21:16:38.505049 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.505017    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6959d868-jbtbm_0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4/telemeter-client/0.log"
Sep 10 21:16:38.528406 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.528362    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6959d868-jbtbm_0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4/reload/0.log"
Sep 10 21:16:38.551480 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.551459    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6959d868-jbtbm_0d62fc2d-2d7f-4b34-879a-30ecd5c6dec4/kube-rbac-proxy/0.log"
Sep 10 21:16:38.595113 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.595082    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f746ccff4-z8sh8_d5f12e0c-0398-41b8-8aea-7b5675fe570b/thanos-query/0.log"
Sep 10 21:16:38.618239 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.618208    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f746ccff4-z8sh8_d5f12e0c-0398-41b8-8aea-7b5675fe570b/kube-rbac-proxy-web/0.log"
Sep 10 21:16:38.640070 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.640039    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f746ccff4-z8sh8_d5f12e0c-0398-41b8-8aea-7b5675fe570b/kube-rbac-proxy/0.log"
Sep 10 21:16:38.660393 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.660348    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f746ccff4-z8sh8_d5f12e0c-0398-41b8-8aea-7b5675fe570b/prom-label-proxy/0.log"
Sep 10 21:16:38.684697 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.684611    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f746ccff4-z8sh8_d5f12e0c-0398-41b8-8aea-7b5675fe570b/kube-rbac-proxy-rules/0.log"
Sep 10 21:16:38.707906 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:38.707850    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f746ccff4-z8sh8_d5f12e0c-0398-41b8-8aea-7b5675fe570b/kube-rbac-proxy-metrics/0.log"
Sep 10 21:16:40.606662 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.606631    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/1.log"
Sep 10 21:16:40.614759 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.614723    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zz76z_c0845dc6-da33-455e-9c95-4c9776458de3/console-operator/2.log"
Sep 10 21:16:40.633047 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.633017    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"]
Sep 10 21:16:40.641962 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.641938    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.646039 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.645454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"]
Sep 10 21:16:40.668416 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.668368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-sys\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.668724 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.668702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-lib-modules\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.668952 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.668918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-proc\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.669163 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.669146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l584\" (UniqueName: \"kubernetes.io/projected/e796c3cf-e473-42bc-8792-473f393fbd28-kube-api-access-4l584\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.669342 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.669319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-podres\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.770484 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.770438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-podres\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.770695 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.770538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-sys\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.770695 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.770580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-lib-modules\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.770695 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.770630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-proc\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.770876 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.770728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l584\" (UniqueName: \"kubernetes.io/projected/e796c3cf-e473-42bc-8792-473f393fbd28-kube-api-access-4l584\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.771283 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.771256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-podres\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.771730 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.771563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-lib-modules\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.771863 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.771611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-sys\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.771955 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.771649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e796c3cf-e473-42bc-8792-473f393fbd28-proc\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.781103 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.781064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l584\" (UniqueName: \"kubernetes.io/projected/e796c3cf-e473-42bc-8792-473f393fbd28-kube-api-access-4l584\") pod \"perf-node-gather-daemonset-dkv9n\" (UID: \"e796c3cf-e473-42bc-8792-473f393fbd28\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:40.958478 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:40.958400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:41.220023 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:41.219938    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c45d6f6dd-lwvjk_9d82c47e-4c69-4b23-b785-3d219141f720/console/0.log"
Sep 10 21:16:41.255647 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:41.255612    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"]
Sep 10 21:16:41.269501 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:41.269432    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-w9l65_4caa1c8e-a1de-4a18-8314-bda6d72b8147/download-server/0.log"
Sep 10 21:16:41.961791 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:41.961760    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-sdfnk_0bc0a8b6-f59d-43b0-b028-d5c604f079f1/volume-data-source-validator/0.log"
Sep 10 21:16:42.015738 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:42.015676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n" event={"ID":"e796c3cf-e473-42bc-8792-473f393fbd28","Type":"ContainerStarted","Data":"baccd98616b3c02f93799ff0a8fc94ed5566bfe8176e1261fa156018af1c020d"}
Sep 10 21:16:42.015936 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:42.015770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n" event={"ID":"e796c3cf-e473-42bc-8792-473f393fbd28","Type":"ContainerStarted","Data":"23dacdb52cae6bd04af11f8c676fbda0bdbf6ce70b8c73e17bd58e54947239a6"}
Sep 10 21:16:42.015936 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:42.015805    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:42.035638 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:42.035578    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n" podStartSLOduration=2.035558657 podStartE2EDuration="2.035558657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:42.033021321 +0000 UTC m=+2437.944058606" watchObservedRunningTime="2026-09-10 21:16:42.035558657 +0000 UTC m=+2437.946595942"
Sep 10 21:16:43.113860 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:43.113827    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hkl9m_b492afb9-cddd-4ef5-a90e-6c89771312a9/dns/0.log"
Sep 10 21:16:43.134112 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:43.134086    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hkl9m_b492afb9-cddd-4ef5-a90e-6c89771312a9/kube-rbac-proxy/0.log"
Sep 10 21:16:43.199405 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:43.199377    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vzjhd_5b91ad48-6bac-4b33-964b-f6cc1c5bada9/dns-node-resolver/0.log"
Sep 10 21:16:43.768219 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:43.768189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-glc9l_c0d0e19d-c65f-4d91-9151-8340d44d2502/node-ca/0.log"
Sep 10 21:16:45.036045 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:45.036014    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-r45j2_4de93ecd-2faf-4b03-9db2-f6cbb829a253/kube-auth-proxy/0.log"
Sep 10 21:16:45.071627 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:45.071590    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-sspzh_62163f71-e629-4980-81da-76382d8e33ff/istio-proxy/0.log"
Sep 10 21:16:45.083438 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:45.083414    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-sspzh_62163f71-e629-4980-81da-76382d8e33ff/istio-proxy/1.log"
Sep 10 21:16:45.188721 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:45.188676    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lq8k2_278ec5f3-c99d-47bb-99d1-0185806fe1e0/payload-processing/0.log"
Sep 10 21:16:46.120107 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:46.120080    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bp5xz_dbfc039d-4acd-4649-a477-b18f2c799a86/serve-healthcheck-canary/0.log"
Sep 10 21:16:46.668639 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:46.668609    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-w5d4j_3992e74c-f2ea-432a-8b36-bdfe17d9766a/insights-operator/1.log"
Sep 10 21:16:46.669389 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:46.669374    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-w5d4j_3992e74c-f2ea-432a-8b36-bdfe17d9766a/insights-operator/0.log"
Sep 10 21:16:46.688483 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:46.688461    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8td6j_b7227f57-b436-4625-bf3a-20086578fde4/kube-rbac-proxy/0.log"
Sep 10 21:16:46.708798 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:46.708774    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8td6j_b7227f57-b436-4625-bf3a-20086578fde4/exporter/0.log"
Sep 10 21:16:46.730010 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:46.729987    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8td6j_b7227f57-b436-4625-bf3a-20086578fde4/extractor/0.log"
Sep 10 21:16:48.031049 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:48.031016    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-dkv9n"
Sep 10 21:16:49.067659 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:49.067626    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fqgdq_36ff9a55-515f-42ff-98c3-b8ada9d6225e/manager/0.log"
Sep 10 21:16:49.297438 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:49.296869    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xsvmm_9910bb83-a139-4222-afa0-d3bee0b2b885/manager/0.log"
Sep 10 21:16:49.724561 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:49.724528    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-xm2qs_2e2ccf4b-2f6d-46fa-bfe0-b18c41007505/manager/0.log"
Sep 10 21:16:49.743814 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:49.743791    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-xm2qs_2e2ccf4b-2f6d-46fa-bfe0-b18c41007505/copy-manifests/0.log"
Sep 10 21:16:51.168349 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:51.168307    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-j48m6_2051e58e-f44c-4860-8fbb-7102ca1c1414/openshift-lws-operator/0.log"
Sep 10 21:16:51.317967 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:51.317920    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-r45j2_4de93ecd-2faf-4b03-9db2-f6cbb829a253/kube-auth-proxy/0.log"
Sep 10 21:16:51.339334 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:51.339295    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-sspzh_62163f71-e629-4980-81da-76382d8e33ff/istio-proxy/0.log"
Sep 10 21:16:51.351517 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:51.351491    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-sspzh_62163f71-e629-4980-81da-76382d8e33ff/istio-proxy/1.log"
Sep 10 21:16:51.416411 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:51.416373    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lq8k2_278ec5f3-c99d-47bb-99d1-0185806fe1e0/payload-processing/0.log"
Sep 10 21:16:53.139927 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:53.139898    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-r45j2_4de93ecd-2faf-4b03-9db2-f6cbb829a253/kube-auth-proxy/0.log"
Sep 10 21:16:53.162142 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:53.162113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-sspzh_62163f71-e629-4980-81da-76382d8e33ff/istio-proxy/0.log"
Sep 10 21:16:53.171636 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:53.171617    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-sspzh_62163f71-e629-4980-81da-76382d8e33ff/istio-proxy/1.log"
Sep 10 21:16:53.243919 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:53.243876    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lq8k2_278ec5f3-c99d-47bb-99d1-0185806fe1e0/payload-processing/0.log"
Sep 10 21:16:56.385804 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:56.385778    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rtbq7_e37f13fb-6890-40de-8fff-043d0e42a7df/migrator/0.log"
Sep 10 21:16:56.406283 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:56.406258    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rtbq7_e37f13fb-6890-40de-8fff-043d0e42a7df/graceful-termination/0.log"
Sep 10 21:16:57.794945 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.794868    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9f7gx_83b7d0e1-6406-4ea2-bb44-8dc301131080/kube-multus/0.log"
Sep 10 21:16:57.822229 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.822206    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6nvt4_a2bde71e-aa70-43ef-8b80-61755a1e7085/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:16:57.845766 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.845713    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6nvt4_a2bde71e-aa70-43ef-8b80-61755a1e7085/egress-router-binary-copy/0.log"
Sep 10 21:16:57.869830 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.869792    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6nvt4_a2bde71e-aa70-43ef-8b80-61755a1e7085/cni-plugins/0.log"
Sep 10 21:16:57.892001 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.891972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6nvt4_a2bde71e-aa70-43ef-8b80-61755a1e7085/bond-cni-plugin/0.log"
Sep 10 21:16:57.915082 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.915046    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6nvt4_a2bde71e-aa70-43ef-8b80-61755a1e7085/routeoverride-cni/0.log"
Sep 10 21:16:57.947446 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.947418    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6nvt4_a2bde71e-aa70-43ef-8b80-61755a1e7085/whereabouts-cni-bincopy/0.log"
Sep 10 21:16:57.973661 ip-10-0-129-103 kubenswrapper[2565]: I0910 21:16:57.973637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6nvt4_a2bde71e-aa70-43ef-8b80-61755a1e7085/whereabouts-cni/0.log"