Sep 07 11:22:46.763700 ip-10-0-129-163 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 11:22:47.108508 ip-10-0-129-163 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 11:22:47.108508 ip-10-0-129-163 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 11:22:47.108508 ip-10-0-129-163 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 11:22:47.108508 ip-10-0-129-163 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 11:22:47.108508 ip-10-0-129-163 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 11:22:47.110987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.110906    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 11:22:47.114813 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114798    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:47.114813 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114813    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114816    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114820    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114823    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114826    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114829    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114832    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114835    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114838    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114841    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114845    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114847    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114850    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114852    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114856    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114858    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114861    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114864    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114866    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114869    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:47.114884 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114871    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114874    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114878    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114882    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114885    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114888    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114891    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114894    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114897    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114899    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114902    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114905    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114907    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114910    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114913    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114915    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114918    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114921    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114923    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:47.115360 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114926    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114928    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114931    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114933    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114938    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114940    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114943    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114945    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114949    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114952    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114955    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114957    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114960    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114962    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114965    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114968    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114971    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114974    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114977    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:47.115834 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114979    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114982    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114985    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114988    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114992    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114994    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114997    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.114999    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115002    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115005    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115007    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115010    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115013    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115015    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115018    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115020    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115023    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115026    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115029    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115031    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:47.116294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115034    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115036    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115038    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115041    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115044    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115046    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115048    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115051    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115054    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115057    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115060    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115064    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115067    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115071    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115074    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115077    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115079    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115082    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115084    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115086    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:47.116815 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115787    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115793    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115798    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115802    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115805    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115808    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115811    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115814    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115816    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115820    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115822    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115825    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115828    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115830    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115833    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115836    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115838    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115841    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115844    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115846    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:47.117296 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115849    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115853    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115856    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115858    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115861    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115864    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115866    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115871    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115874    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115878    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115880    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115883    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115885    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115888    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115890    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115893    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115895    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115898    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115902    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:47.117809 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115905    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115909    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115912    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115914    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115918    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115921    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115923    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115926    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115928    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115931    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115933    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115936    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115939    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115942    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115944    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115947    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115949    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115952    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115959    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115962    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:47.118274 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115964    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115967    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115969    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115972    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115974    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115977    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115979    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115982    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115984    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115987    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115989    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115991    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115994    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.115998    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116001    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116004    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116006    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116009    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116011    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116014    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:47.118768 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116016    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116019    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116022    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116025    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116028    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116031    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116033    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116036    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116038    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116042    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116044    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116047    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116049    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116052    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116054    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116057    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116059    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116062    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116064    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116067    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:47.119258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116446    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116452    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116455    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116457    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116462    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116466    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116470    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116472    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116475    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116478    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116481    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116484    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116487    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116490    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116492    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116495    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116499    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116502    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116504    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116507    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:47.119769 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116510    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116513    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116515    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116518    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116520    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116523    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116526    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116528    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116531    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116533    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116536    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116539    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116542    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116545    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116547    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116550    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116553    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116560    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116563    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:47.120258 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116566    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116568    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116571    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116573    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116576    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116578    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116581    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116583    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116586    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116590    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116592    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116595    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116597    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116600    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116603    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116605    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116608    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116610    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116613    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:47.120734 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116617    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116620    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116623    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116626    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116628    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116631    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116633    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116636    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116638    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116641    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116643    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116646    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116650    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116653    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116655    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116658    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116660    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116663    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116666    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116668    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:47.121201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116670    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116673    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116677    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116679    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116682    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116685    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116687    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116691    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116693    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116696    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116699    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116701    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116705    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116709    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116712    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116715    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116718    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116721    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116724    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116726    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:47.121740 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.116729    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116810    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116843    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116850    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116854    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116859    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116863    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116867    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116872    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116875    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116878    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116882    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116885    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116888    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116892    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116895    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116898    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116901    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116904    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116908    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116911    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116914    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116917    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116920    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 11:22:47.122236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116924    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116927    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116930    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116933    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116936    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116939    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116942    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116945    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116948    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116952    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116954    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116957    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116960    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116963    2578 flags.go:64] FLAG: --enable-server="true"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116966    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116974    2578 flags.go:64] FLAG: --event-burst="100"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116977    2578 flags.go:64] FLAG: --event-qps="50"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116980    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116983    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116986    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116990    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116993    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.116997    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117000    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117002    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 11:22:47.122837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117005    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117008    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117011    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117014    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117017    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117020    2578 flags.go:64] FLAG: --feature-gates=""
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117023    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117026    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117029    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117032    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117035    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117038    2578 flags.go:64] FLAG: --help="false"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117041    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117044    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117047    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117051    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117054    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117057    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117060    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117063    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117066    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117068    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117077    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117080    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 11:22:47.123461 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117083    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117087    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117089    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117092    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117095    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117099    2578 flags.go:64] FLAG: --lock-file=""
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117102    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117105    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117108    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117113    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117116    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117118    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117121    2578 flags.go:64] FLAG: --logging-format="text"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117124    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117127    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117130    2578 flags.go:64] FLAG: --manifest-url=""
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117133    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117138    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117140    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117144    2578 flags.go:64] FLAG: --max-pods="110"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117147    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117150    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117153    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117156    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117159    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 11:22:47.124149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117162    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117164    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117171    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117173    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117176    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117181    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117191    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117196    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117199    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117202    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117205    2578 flags.go:64] FLAG: --port="10250"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117208    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117211    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e60d199b952a9a19"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117215    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117218    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117221    2578 flags.go:64] FLAG: --register-node="true"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117224    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117228    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117231    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117233    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117236    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117239    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117243    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117246    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117248    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 11:22:47.125332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117251    2578 flags.go:64] FLAG: --runonce="false"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117254    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117257    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117260    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117263    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117265    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117268    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117271    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117274    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117277    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117280    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117282    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117287    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117289    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117298    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117301    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117306    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117309    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117312    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117318    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117321    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117324    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117327    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117330    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117332    2578 flags.go:64] FLAG: --v="2"
Sep 07 11:22:47.126465 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117336    2578 flags.go:64] FLAG: --version="false"
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117340    2578 flags.go:64] FLAG: --vmodule=""
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117344    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.117347    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117507    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117513    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117516    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117519    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117522    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117525    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117529    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117532    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117534    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117537    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117540    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117545    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117548    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117551    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117554    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:47.127668 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117558    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117561    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117563    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117572    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117574    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117577    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117580    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117582    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117585    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117588    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117591    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117594    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117596    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117599    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117601    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117604    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117607    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117609    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117612    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117615    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:47.128456 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117618    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117620    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117623    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117626    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117628    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117631    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117633    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117636    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117639    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117641    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117644    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117646    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117651    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117653    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117656    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117658    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117666    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117669    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117672    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117674    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:47.128963 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117677    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117679    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117682    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117684    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117687    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117689    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117692    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117700    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117703    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117705    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117708    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117711    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117714    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117717    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117720    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117724    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117726    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117729    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117733    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:47.129816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117736    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117739    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117742    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117745    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117748    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117751    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117754    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117757    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117760    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117762    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117771    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117774    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117776    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117779    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117782    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117785    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117788    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117790    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117793    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:47.130581 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117796    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117799    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117801    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117804    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117807    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.117809    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.118442    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.126678    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.126697    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.126723    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.126737    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126848    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126857    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126864    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:47.131048 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126870    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126875    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126879    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126884    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126889    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126893    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126898    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126903    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126907    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126912    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126916    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126920    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126925    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126929    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126933    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126937    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126941    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126945    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126950    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:47.131600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126954    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126959    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126963    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126967    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126971    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126975    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126980    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126993    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.126998    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127002    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127007    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127011    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127015    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127019    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127023    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127028    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127032    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127036    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127041    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:47.132081 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127045    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127049    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127053    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127057    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127061    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127066    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127070    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127074    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127078    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127082    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127086    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127090    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127094    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127099    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127103    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127107    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127111    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127115    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127119    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127124    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:47.132615 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127128    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127143    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127147    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127151    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127158    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127163    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127168    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127172    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127177    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127182    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127188    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127193    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127198    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127203    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127208    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127214    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127219    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127224    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127228    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:47.133145 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127233    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127237    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127241    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127245    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127249    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127254    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127258    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127262    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127266    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127270    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127274    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127279    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127283    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127288    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127293    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127305    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127310    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127314    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:47.133710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127318    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.127326    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127570    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127581    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127593    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127597    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127602    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127606    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127611    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127615    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127619    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127623    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127627    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127632    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127636    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:47.134285 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127640    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127644    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127648    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127652    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127657    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127661    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127665    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127669    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127672    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127677    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127681    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127685    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127689    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127694    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127699    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127711    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127716    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127720    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127724    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127729    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:47.134924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127733    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127737    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127743    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127751    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127755    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127759    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127763    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127767    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127772    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127776    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127780    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127784    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127789    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127795    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127801    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127805    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127811    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127816    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127820    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:47.135638 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127824    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127829    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127834    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127838    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127842    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127846    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127851    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127855    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127859    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127870    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127875    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127879    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127883    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127888    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127892    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127896    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127900    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127904    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127908    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:47.136098 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127912    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127918    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127923    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127928    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127933    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127937    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127942    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127946    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127950    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127954    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127958    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127962    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127966    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127970    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127974    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127978    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127983    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127987    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127991    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:47.136600 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127995    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.127999    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.128004    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.128008    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.128024    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.128028    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.128033    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.128037    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:47.128041    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.128049    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.128228    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.131173    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.132059    2578 server.go:1019] "Starting client certificate rotation"
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.132163    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 11:22:47.137059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.132220    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 11:22:47.137532 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.134795    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 11:22:47.140439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.140417    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 11:22:47.151458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.151439    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 07 11:22:47.155210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.155191    2578 log.go:25] "Validated CRI v1 image API"
Sep 07 11:22:47.156957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.156941    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 11:22:47.158981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.158949    2578 fs.go:135] Filesystem UUIDs: map[1d16362a-73a4-467d-8c88-304d5c5bc85f:/dev/nvme0n1p4 4d619afe-256d-402b-8ce0-3203e28caa70:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 07 11:22:47.159094 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.158982    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 11:22:47.161847 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.161830    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 11:22:47.164104 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.163994    2578 manager.go:216] Machine: {Timestamp:2026-09-07 11:22:47.162803932 +0000 UTC m=+0.295722041 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102538 MemoryCapacity:33164455936 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec288d5a16c90b53edc07dac4d283c09 SystemUUID:ec288d5a-16c9-0b53-edc0-7dac4d283c09 BootID:c048fef5-5f4a-44a1-8f21-3a771736ec3b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582225920 Type:vfs Inodes:4048395 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582230016 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:33:40:79:a0:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:33:40:79:a0:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:33:ce:d4:ce:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164455936 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 11:22:47.164104 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.164103    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 11:22:47.164201 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.164180    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 11:22:47.167020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.166997    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 11:22:47.167157 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.167023    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 11:22:47.167742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.167731    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 11:22:47.167774 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.167744    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 11:22:47.167774 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.167751    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 11:22:47.168363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.168347    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 11:22:47.169369 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.169359    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 11:22:47.173991 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.173979    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 11:22:47.174106 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174097    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 11:22:47.174144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174137    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 11:22:47.174173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174147    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 11:22:47.174173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174161    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 11:22:47.174240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174192    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 07 11:22:47.174240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174209    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 11:22:47.174596 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174583    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 11:22:47.174637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.174609    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 11:22:47.176538 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.176518    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 11:22:47.179087 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.179073    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 11:22:47.179152 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.179102    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 11:22:47.180090 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180078    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180093    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180100    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180105    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180114    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180119    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180133    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180138    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180146    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 11:22:47.180149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180151    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 11:22:47.180547 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180159    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 11:22:47.180547 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180164    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 11:22:47.180754 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180734    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 11:22:47.180754 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.180745    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 11:22:47.182336 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.182302    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 11:22:47.182437 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.182332    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 11:22:47.184248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.184233    2578 server.go:1268] "Started kubelet"
Sep 07 11:22:47.185586 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.185539    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 11:22:47.185975 ip-10-0-129-163 systemd[1]: Started Kubernetes Kubelet.
Sep 07 11:22:47.186180 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.185901    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 11:22:47.186180 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.186082    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 11:22:47.188467 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.188435    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsz57"
Sep 07 11:22:47.189570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.189522    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 11:22:47.189997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.189979    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 11:22:47.190654 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.190631    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 11:22:47.193405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.193379    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 11:22:47.194022 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.193187    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-163.ec2.internal.18d30615ae474390  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-163.ec2.internal,UID:ip-10-0-129-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-163.ec2.internal,},FirstTimestamp:2026-09-07 11:22:47.184204688 +0000 UTC m=+0.317122775,LastTimestamp:2026-09-07 11:22:47.184204688 +0000 UTC m=+0.317122775,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-163.ec2.internal,}"
Sep 07 11:22:47.196108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.196088    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 11:22:47.196659 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.196635    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 11:22:47.196830 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.196809    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.197349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197324    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 11:22:47.197349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197344    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 11:22:47.197534 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197324    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 11:22:47.197534 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197481    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 11:22:47.197534 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197490    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 11:22:47.197655 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197605    2578 factory.go:153] Registering CRI-O factory
Sep 07 11:22:47.197655 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197619    2578 factory.go:223] Registration of the crio container factory successfully
Sep 07 11:22:47.197726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197659    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 11:22:47.197726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197665    2578 factory.go:55] Registering systemd factory
Sep 07 11:22:47.197726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197671    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 07 11:22:47.197726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197682    2578 factory.go:103] Registering Raw factory
Sep 07 11:22:47.197726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197692    2578 manager.go:1196] Started watching for new ooms in manager
Sep 07 11:22:47.197966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.197956    2578 manager.go:349] Starting recovery of all containers
Sep 07 11:22:47.199246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.199224    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsz57"
Sep 07 11:22:47.200867 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.200826    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 11:22:47.200965 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.200942    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 11:22:47.206575 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.206558    2578 manager.go:354] Recovery completed
Sep 07 11:22:47.210811 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.210710    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:47.212323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.212309    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:47.212424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.212343    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:47.212424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.212357    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:47.213958 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.213937    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 11:22:47.214044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.213964    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 11:22:47.214044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.213982    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 11:22:47.214845 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.214825    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:47.215029 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.214944    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-163.ec2.internal.18d30615aff452dc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-163.ec2.internal,UID:ip-10-0-129-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-163.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-163.ec2.internal,},FirstTimestamp:2026-09-07 11:22:47.212323548 +0000 UTC m=+0.345241638,LastTimestamp:2026-09-07 11:22:47.212323548 +0000 UTC m=+0.345241638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-163.ec2.internal,}"
Sep 07 11:22:47.216290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.216274    2578 policy_none.go:49] "None policy: Start"
Sep 07 11:22:47.216361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.216295    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 11:22:47.216361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.216311    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 11:22:47.217296 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.217279    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:47.217375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.217311    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:47.217375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.217326    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:47.218423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.218407    2578 policy_none.go:47] "Start"
Sep 07 11:22:47.294133 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.294110    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 11:22:47.296172 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.295195    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 11:22:47.296172 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.295251    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 11:22:47.296172 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.295277    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 11:22:47.296172 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.295353    2578 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 07 11:22:47.296999 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.296973    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.297316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.297300    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:47.391995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.391981    2578 manager.go:337] "Starting Device Plugin manager"
Sep 07 11:22:47.392081 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.392016    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 11:22:47.392081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.392026    2578 server.go:87] "Starting device plugin registration server"
Sep 07 11:22:47.392433 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.392419    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 11:22:47.392486 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.392438    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 11:22:47.392538 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.392516    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 11:22:47.392633 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.392616    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 11:22:47.392633 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.392630    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 11:22:47.393201 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.393180    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 11:22:47.393280 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.393239    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.395514 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.395495    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"]
Sep 07 11:22:47.395596 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.395567    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:47.396901 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.396886    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:47.396967 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.396911    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:47.396967 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.396922    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:47.399155 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399141    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:47.399294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.399351 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399316    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:47.399846 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399829    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:47.399920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399860    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:47.399920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399872    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:47.399920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399829    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:47.400011 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399936    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:47.400011 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.399949    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:47.401978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.401960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.402065 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.401983    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:47.402626 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.402613    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:47.402709 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.402636    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:47.402709 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.402646    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:47.410850 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.410833    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.422915 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.422890    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.426987 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.426968    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.493094 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.493076    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:47.494143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.494129    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:47.494202 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.494153    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:47.494202 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.494162    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:47.494202 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.494180    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.498792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.498772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fda4d66429fdb419c55741b8bfee0a5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"fda4d66429fdb419c55741b8bfee0a5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.498874 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.498797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda4d66429fdb419c55741b8bfee0a5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"fda4d66429fdb419c55741b8bfee0a5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.498874 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.498813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07877c340319b68616068533f7d8229b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"07877c340319b68616068533f7d8229b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.499730 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.499716    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.499795 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.499740    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-163.ec2.internal\": node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.515172 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.515153    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.599721 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.599691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fda4d66429fdb419c55741b8bfee0a5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"fda4d66429fdb419c55741b8bfee0a5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.599819 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.599731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda4d66429fdb419c55741b8bfee0a5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"fda4d66429fdb419c55741b8bfee0a5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.599819 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.599769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07877c340319b68616068533f7d8229b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"07877c340319b68616068533f7d8229b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.599819 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.599792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fda4d66429fdb419c55741b8bfee0a5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"fda4d66429fdb419c55741b8bfee0a5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.599929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.599823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda4d66429fdb419c55741b8bfee0a5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"fda4d66429fdb419c55741b8bfee0a5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.599929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.599824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07877c340319b68616068533f7d8229b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"07877c340319b68616068533f7d8229b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.615848 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.615802    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.716595 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.716574    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.725741 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.725724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.731701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:47.731686    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:47.816871 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.816847    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:47.917416 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:47.917348    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:48.017824 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:48.017804    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:48.118272 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:48.118251    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:48.132439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:48.132417    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 11:22:48.132601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:48.132580    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 11:22:48.204739 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:48.204673    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 11:17:48 +0000 UTC" deadline="2028-06-01 02:43:53.278226424 +0000 UTC"
Sep 07 11:22:48.204739 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:48.204699    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15183h21m5.073530499s"
Sep 07 11:22:48.219107 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:48.219078    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:49.918674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:49.918655    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:49.950806 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:49.950774    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d66429fdb419c55741b8bfee0a5b.slice/crio-15a0d19b2c500c7573fa48f9595f539be26ff487268d4c678eb7df614a653b48 WatchSource:0}: Error finding container 15a0d19b2c500c7573fa48f9595f539be26ff487268d4c678eb7df614a653b48: Status 404 returned error can't find the container with id 15a0d19b2c500c7573fa48f9595f539be26ff487268d4c678eb7df614a653b48
Sep 07 11:22:49.951126 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:49.951106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07877c340319b68616068533f7d8229b.slice/crio-355ca54427d679e1cee15ae4cb004d5ed36a693d001defa7ae138c409e78e010 WatchSource:0}: Error finding container 355ca54427d679e1cee15ae4cb004d5ed36a693d001defa7ae138c409e78e010: Status 404 returned error can't find the container with id 355ca54427d679e1cee15ae4cb004d5ed36a693d001defa7ae138c409e78e010
Sep 07 11:22:49.956638 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:49.956617    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:22:49.956739 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:49.956647    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"fda4d66429fdb419c55741b8bfee0a5b","Type":"ContainerStarted","Data":"15a0d19b2c500c7573fa48f9595f539be26ff487268d4c678eb7df614a653b48"}
Sep 07 11:22:49.959546 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:49.959520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" event={"ID":"07877c340319b68616068533f7d8229b","Type":"ContainerStarted","Data":"355ca54427d679e1cee15ae4cb004d5ed36a693d001defa7ae138c409e78e010"}
Sep 07 11:22:49.977388 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:49.977368    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:50.078013 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.077971    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:50.178566 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.178512    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:50.201118 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.201099    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:50.279567 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.279539    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:50.380389 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.380357    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Sep 07 11:22:50.430801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.430596    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:50.455826 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.455792    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:50.466513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.466485    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 11:22:50.467565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.467542    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Sep 07 11:22:50.476158 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.476135    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 11:22:50.758976 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.758898    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:50.833175 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.833141    2578 apiserver.go:52] "Watching apiserver"
Sep 07 11:22:50.842025 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.842001    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 11:22:50.843857 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.843829    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-sjsts","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq","openshift-cluster-node-tuning-operator/tuned-ms8n9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal","openshift-multus/multus-additional-cni-plugins-bwz6m","openshift-multus/network-metrics-daemon-lthxp","openshift-network-diagnostics/network-check-target-kjcnz","openshift-network-operator/iptables-alerter-flng9","kube-system/konnectivity-agent-47js8","kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal","openshift-dns/node-resolver-hkpwd","openshift-image-registry/node-ca-vnkxf","openshift-multus/multus-k7j9h","openshift-ovn-kubernetes/ovnkube-node-br8kb"]
Sep 07 11:22:50.846318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.846299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.848531 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.848509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.849012 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.848989    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 11:22:50.849105 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.848990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 11:22:50.849105 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.849078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rdtd4\""
Sep 07 11:22:50.849105 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.849087    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 11:22:50.849258 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.849200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 11:22:50.850673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.850604    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 11:22:50.850823 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.850787    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 11:22:50.850909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.850826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.850909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.850855    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 11:22:50.851070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.850962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gtd26\""
Sep 07 11:22:50.852993 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.852974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h8gjf\""
Sep 07 11:22:50.853125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.853042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.853125 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.853113    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:22:50.853257 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.853213    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 11:22:50.853318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.853262    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:22:50.854759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.854736    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 11:22:50.855140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.855113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.857323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.857246    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 11:22:50.857323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.857300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bv9nt\""
Sep 07 11:22:50.857481 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.857303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:50.857481 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.857439    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:22:50.857565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.857523    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 11:22:50.859647 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.859630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:50.859738 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.859689    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:22:50.864215 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.863871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:50.866145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.866125    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:22:50.866256 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.866172    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 11:22:50.866256 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.866250    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j27fp\""
Sep 07 11:22:50.866441 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.866422    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 11:22:50.866972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.866717    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 11:22:50.866972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.866719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:50.867120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.866995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-kubelet-config\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.867120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.867120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.867120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-k8s-cni-cncf-io\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.867120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-cni-multus\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.867120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.867120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mwp\" (UniqueName: \"kubernetes.io/projected/46731aa1-8800-4c5a-883a-563959a51a3c-kube-api-access-h6mwp\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-var-lib-kubelet\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-cnibin\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-cni-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-device-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-modprobe-d\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysconfig\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysctl-d\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-lib-modules\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-hostroot\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-conf-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-multus-certs\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-kubernetes\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.867541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-run\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-tmp\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86vq\" (UniqueName: \"kubernetes.io/projected/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-kube-api-access-k86vq\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-netns\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-cni-bin\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-etc-kubernetes\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-systemd\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-cnibin\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2cnl\" (UniqueName: \"kubernetes.io/projected/006ca005-7640-42ec-86c4-7a55ca8ff076-kube-api-access-n2cnl\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-tuned\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.868255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.867991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgmg\" (UniqueName: \"kubernetes.io/projected/a90bddad-de76-4653-8a1b-e3546fb5aa7f-kube-api-access-4dgmg\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-system-cni-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/006ca005-7640-42ec-86c4-7a55ca8ff076-cni-binary-copy\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-socket-dir-parent\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-sys\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-dbus\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjvd\" (UniqueName: \"kubernetes.io/projected/4dba6b53-96bc-4321-8260-caa270ad6ca9-kube-api-access-ggjvd\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-kubelet\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-daemon-config\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysctl-conf\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-host\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t2m96\""
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.868903    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 11:22:50.869219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.869040    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 11:22:50.870008 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.869260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.871629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.871612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:50.872019 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.872000    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 11:22:50.872111 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.872056    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 11:22:50.872111 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.872065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 11:22:50.872111 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.872083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 11:22:50.872111 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.872092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 11:22:50.872290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.872185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 11:22:50.872346 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.872311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8kv7f\""
Sep 07 11:22:50.873743 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.873725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wqspj\""
Sep 07 11:22:50.873891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.873827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:50.873963 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.873924    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 11:22:50.874355 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.874339    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 11:22:50.876070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.876050    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z9frs\""
Sep 07 11:22:50.876164 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.876050    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 11:22:50.876164 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.876087    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 11:22:50.876268 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.876254    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 11:22:50.888007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.887989    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l588g"
Sep 07 11:22:50.896421 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.896403    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l588g"
Sep 07 11:22:50.956246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.956224    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 11:22:50.968964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.968938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fe92fb-f492-45be-894d-7e4428d1985b-tmp-dir\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:50.969074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.968975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:50.969074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-system-cni-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.969074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjvd\" (UniqueName: \"kubernetes.io/projected/4dba6b53-96bc-4321-8260-caa270ad6ca9-kube-api-access-ggjvd\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.969074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-kubelet\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.969074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-kubelet-config\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.969349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.969349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-slash\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.969349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-system-cni-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.969349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-kubelet-config\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.969349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-kubelet\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.969349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-systemd\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.969349 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.969348    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-run-ovn-kubernetes\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-cni-bin\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.969419    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret podName:3a1df1aa-6e57-4a72-8f6f-06c8588fab09 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:51.469403018 +0000 UTC m=+2.944251149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret") pod "global-pull-secret-syncer-sjsts" (UID: "3a1df1aa-6e57-4a72-8f6f-06c8588fab09") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-k8s-cni-cncf-io\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-cnibin\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-k8s-cni-cncf-io\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-device-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-modprobe-d\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/008e2c25-f1aa-440f-9693-2e696410ef10-host-slash\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-cnibin\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-run-netns\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-modprobe-d\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.969712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jcgp\" (UniqueName: \"kubernetes.io/projected/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-kube-api-access-9jcgp\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqff\" (UniqueName: \"kubernetes.io/projected/50fe92fb-f492-45be-894d-7e4428d1985b-kube-api-access-qfqff\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-device-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-hostroot\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-conf-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-hostroot\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-multus-certs\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-conf-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-kubernetes\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.969972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-run\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-cni-netd\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-cni-bin\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-kubernetes\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-etc-kubernetes\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-multus-certs\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-systemd\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-run\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.970414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-etc-kubernetes\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff53ccaf-e144-41af-8248-489446634ebd-host\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-cni-bin\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2cnl\" (UniqueName: \"kubernetes.io/projected/006ca005-7640-42ec-86c4-7a55ca8ff076-kube-api-access-n2cnl\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovnkube-config\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rsx\" (UniqueName: \"kubernetes.io/projected/008e2c25-f1aa-440f-9693-2e696410ef10-kube-api-access-n5rsx\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-systemd-units\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-etc-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/006ca005-7640-42ec-86c4-7a55ca8ff076-cni-binary-copy\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-socket-dir-parent\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-sys\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-dbus\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.971267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-socket-dir-parent\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-ovn\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-daemon-config\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysctl-conf\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-host\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-dbus\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-systemd\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32d447c3-1090-4e73-a7a4-c510811c26e3-konnectivity-ca\") pod \"konnectivity-agent-47js8\" (UID: \"32d447c3-1090-4e73-a7a4-c510811c26e3\") " pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-sys\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-cni-multus\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-var-lib-cni-multus\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mwp\" (UniqueName: \"kubernetes.io/projected/46731aa1-8800-4c5a-883a-563959a51a3c-kube-api-access-h6mwp\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-var-lib-kubelet\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-log-socket\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-cni-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.972036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysconfig\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysctl-conf\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysctl-d\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-lib-modules\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-host\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovnkube-script-lib\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.970927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/006ca005-7640-42ec-86c4-7a55ca8ff076-cni-binary-copy\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dba6b53-96bc-4321-8260-caa270ad6ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-cni-dir\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysconfig\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-var-lib-kubelet\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-sysctl-d\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-lib-modules\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-tmp\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.972751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86vq\" (UniqueName: \"kubernetes.io/projected/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-kube-api-access-k86vq\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5drtd\" (UniqueName: \"kubernetes.io/projected/ff53ccaf-e144-41af-8248-489446634ebd-kube-api-access-5drtd\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-node-log\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-env-overrides\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-netns\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff53ccaf-e144-41af-8248-489446634ebd-serviceca\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32d447c3-1090-4e73-a7a4-c510811c26e3-agent-certs\") pod \"konnectivity-agent-47js8\" (UID: \"32d447c3-1090-4e73-a7a4-c510811c26e3\") " pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/006ca005-7640-42ec-86c4-7a55ca8ff076-multus-daemon-config\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-kubelet\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-host-run-netns\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.971741    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovn-node-metrics-cert\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:50.971793    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:22:51.471780165 +0000 UTC m=+2.946628292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-cnibin\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.973493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/006ca005-7640-42ec-86c4-7a55ca8ff076-cnibin\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-tuned\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-var-lib-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgmg\" (UniqueName: \"kubernetes.io/projected/a90bddad-de76-4653-8a1b-e3546fb5aa7f-kube-api-access-4dgmg\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971966    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.971988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/008e2c25-f1aa-440f-9693-2e696410ef10-iptables-alerter-script\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.972040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46731aa1-8800-4c5a-883a-563959a51a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.972075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50fe92fb-f492-45be-894d-7e4428d1985b-hosts-file\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:50.974288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.972410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4dba6b53-96bc-4321-8260-caa270ad6ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.975486 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.975454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-tmp\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.975579 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.975515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-etc-tuned\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.983221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.983197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjvd\" (UniqueName: \"kubernetes.io/projected/4dba6b53-96bc-4321-8260-caa270ad6ca9-kube-api-access-ggjvd\") pod \"multus-additional-cni-plugins-bwz6m\" (UID: \"4dba6b53-96bc-4321-8260-caa270ad6ca9\") " pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:50.983759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.983739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mwp\" (UniqueName: \"kubernetes.io/projected/46731aa1-8800-4c5a-883a-563959a51a3c-kube-api-access-h6mwp\") pod \"aws-ebs-csi-driver-node-qnfrq\" (UID: \"46731aa1-8800-4c5a-883a-563959a51a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:50.983847 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.983811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2cnl\" (UniqueName: \"kubernetes.io/projected/006ca005-7640-42ec-86c4-7a55ca8ff076-kube-api-access-n2cnl\") pod \"multus-k7j9h\" (UID: \"006ca005-7640-42ec-86c4-7a55ca8ff076\") " pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:50.984299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.984268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86vq\" (UniqueName: \"kubernetes.io/projected/bf5c7178-3e5d-4ddc-aac9-e7406507d9bb-kube-api-access-k86vq\") pod \"tuned-ms8n9\" (UID: \"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:50.984629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:50.984604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgmg\" (UniqueName: \"kubernetes.io/projected/a90bddad-de76-4653-8a1b-e3546fb5aa7f-kube-api-access-4dgmg\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:51.073269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-systemd-units\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-etc-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-ovn\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-ovn\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-systemd-units\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-etc-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32d447c3-1090-4e73-a7a4-c510811c26e3-konnectivity-ca\") pod \"konnectivity-agent-47js8\" (UID: \"32d447c3-1090-4e73-a7a4-c510811c26e3\") " pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-log-socket\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovnkube-script-lib\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drtd\" (UniqueName: \"kubernetes.io/projected/ff53ccaf-e144-41af-8248-489446634ebd-kube-api-access-5drtd\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-node-log\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073507    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-log-socket\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.073570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-env-overrides\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff53ccaf-e144-41af-8248-489446634ebd-serviceca\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32d447c3-1090-4e73-a7a4-c510811c26e3-agent-certs\") pod \"konnectivity-agent-47js8\" (UID: \"32d447c3-1090-4e73-a7a4-c510811c26e3\") " pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-kubelet\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovn-node-metrics-cert\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-var-lib-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/008e2c25-f1aa-440f-9693-2e696410ef10-iptables-alerter-script\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50fe92fb-f492-45be-894d-7e4428d1985b-hosts-file\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fe92fb-f492-45be-894d-7e4428d1985b-tmp-dir\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-node-log\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-slash\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-systemd\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-run-ovn-kubernetes\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-slash\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-systemd\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-kubelet\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.073999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-run-ovn-kubernetes\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32d447c3-1090-4e73-a7a4-c510811c26e3-konnectivity-ca\") pod \"konnectivity-agent-47js8\" (UID: \"32d447c3-1090-4e73-a7a4-c510811c26e3\") " pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50fe92fb-f492-45be-894d-7e4428d1985b-hosts-file\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-cni-bin\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-cni-bin\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/008e2c25-f1aa-440f-9693-2e696410ef10-host-slash\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-run-netns\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jcgp\" (UniqueName: \"kubernetes.io/projected/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-kube-api-access-9jcgp\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqff\" (UniqueName: \"kubernetes.io/projected/50fe92fb-f492-45be-894d-7e4428d1985b-kube-api-access-qfqff\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-cni-netd\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff53ccaf-e144-41af-8248-489446634ebd-host\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovnkube-config\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovnkube-script-lib\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rsx\" (UniqueName: \"kubernetes.io/projected/008e2c25-f1aa-440f-9693-2e696410ef10-kube-api-access-n5rsx\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:51.074812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-var-lib-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/008e2c25-f1aa-440f-9693-2e696410ef10-iptables-alerter-script\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff53ccaf-e144-41af-8248-489446634ebd-host\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff53ccaf-e144-41af-8248-489446634ebd-serviceca\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-run-openvswitch\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/008e2c25-f1aa-440f-9693-2e696410ef10-host-slash\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-run-netns\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-env-overrides\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-host-cni-netd\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovnkube-config\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.075602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.074985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fe92fb-f492-45be-894d-7e4428d1985b-tmp-dir\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:51.076650 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.076622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-ovn-node-metrics-cert\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.076765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.076733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32d447c3-1090-4e73-a7a4-c510811c26e3-agent-certs\") pod \"konnectivity-agent-47js8\" (UID: \"32d447c3-1090-4e73-a7a4-c510811c26e3\") " pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:51.084666 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.084614    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:51.084666 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.084641    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:51.084666 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.084655    2578 projected.go:196] Error preparing data for projected volume kube-api-access-l7zlp for pod openshift-network-diagnostics/network-check-target-kjcnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:51.084872 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.084732    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp podName:91fce2a6-f2fd-4d41-9a30-e91037d9dc3b nodeName:}" failed. No retries permitted until 2026-09-07 11:22:51.584714268 +0000 UTC m=+3.059562409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l7zlp" (UniqueName: "kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp") pod "network-check-target-kjcnz" (UID: "91fce2a6-f2fd-4d41-9a30-e91037d9dc3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:51.086354 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.086331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drtd\" (UniqueName: \"kubernetes.io/projected/ff53ccaf-e144-41af-8248-489446634ebd-kube-api-access-5drtd\") pod \"node-ca-vnkxf\" (UID: \"ff53ccaf-e144-41af-8248-489446634ebd\") " pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:51.087084 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.087061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rsx\" (UniqueName: \"kubernetes.io/projected/008e2c25-f1aa-440f-9693-2e696410ef10-kube-api-access-n5rsx\") pod \"iptables-alerter-flng9\" (UID: \"008e2c25-f1aa-440f-9693-2e696410ef10\") " pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:51.087597 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.087572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jcgp\" (UniqueName: \"kubernetes.io/projected/aaabcd49-ae2b-4838-8cdf-4cf86ace72de-kube-api-access-9jcgp\") pod \"ovnkube-node-br8kb\" (UID: \"aaabcd49-ae2b-4838-8cdf-4cf86ace72de\") " pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.087701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.087601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqff\" (UniqueName: \"kubernetes.io/projected/50fe92fb-f492-45be-894d-7e4428d1985b-kube-api-access-qfqff\") pod \"node-resolver-hkpwd\" (UID: \"50fe92fb-f492-45be-894d-7e4428d1985b\") " pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:51.157479 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.157430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7j9h"
Sep 07 11:22:51.166485 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.166453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq"
Sep 07 11:22:51.175115 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.175096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ms8n9"
Sep 07 11:22:51.181662 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.181636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwz6m"
Sep 07 11:22:51.188276 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.188259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-flng9"
Sep 07 11:22:51.195666 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.195650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:22:51.201201 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.201184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:22:51.210755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.210738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hkpwd"
Sep 07 11:22:51.215269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.215252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnkxf"
Sep 07 11:22:51.477851 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.477751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:51.477851 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.477839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:51.478004 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.477907    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:51.478004 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.477926    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:51.478004 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.477969    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret podName:3a1df1aa-6e57-4a72-8f6f-06c8588fab09 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:52.477954705 +0000 UTC m=+3.952802864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret") pod "global-pull-secret-syncer-sjsts" (UID: "3a1df1aa-6e57-4a72-8f6f-06c8588fab09") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:51.478004 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.477983    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:22:52.477977263 +0000 UTC m=+3.952825390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:51.679314 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.679283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:51.679475 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.679441    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:51.679524 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.679480    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:51.679524 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.679492    2578 projected.go:196] Error preparing data for projected volume kube-api-access-l7zlp for pod openshift-network-diagnostics/network-check-target-kjcnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:51.679581 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:51.679547    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp podName:91fce2a6-f2fd-4d41-9a30-e91037d9dc3b nodeName:}" failed. No retries permitted until 2026-09-07 11:22:52.6795315 +0000 UTC m=+4.154379627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7zlp" (UniqueName: "kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp") pod "network-check-target-kjcnz" (UID: "91fce2a6-f2fd-4d41-9a30-e91037d9dc3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:51.789011 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.788851    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d447c3_1090_4e73_a7a4_c510811c26e3.slice/crio-8bcac4e48a300087fa7dd66d0cee9c3bca1c8d690d1709901b019e66cd78787b WatchSource:0}: Error finding container 8bcac4e48a300087fa7dd66d0cee9c3bca1c8d690d1709901b019e66cd78787b: Status 404 returned error can't find the container with id 8bcac4e48a300087fa7dd66d0cee9c3bca1c8d690d1709901b019e66cd78787b
Sep 07 11:22:51.791009 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.790872    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff53ccaf_e144_41af_8248_489446634ebd.slice/crio-5f1c6daad32c08bcca4bbe75f8915531ee4af15209eb7ebcdb9760580201045d WatchSource:0}: Error finding container 5f1c6daad32c08bcca4bbe75f8915531ee4af15209eb7ebcdb9760580201045d: Status 404 returned error can't find the container with id 5f1c6daad32c08bcca4bbe75f8915531ee4af15209eb7ebcdb9760580201045d
Sep 07 11:22:51.793174 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.793145    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dba6b53_96bc_4321_8260_caa270ad6ca9.slice/crio-de498ac2cacf4e01a0c4295f855047ab31f11bdc9ffd1bcea0759229aef131ff WatchSource:0}: Error finding container de498ac2cacf4e01a0c4295f855047ab31f11bdc9ffd1bcea0759229aef131ff: Status 404 returned error can't find the container with id de498ac2cacf4e01a0c4295f855047ab31f11bdc9ffd1bcea0759229aef131ff
Sep 07 11:22:51.793837 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.793729    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fe92fb_f492_45be_894d_7e4428d1985b.slice/crio-db45123b7eeead180f719d595bbe6229840dad954d4511a35169a04b5d45e326 WatchSource:0}: Error finding container db45123b7eeead180f719d595bbe6229840dad954d4511a35169a04b5d45e326: Status 404 returned error can't find the container with id db45123b7eeead180f719d595bbe6229840dad954d4511a35169a04b5d45e326
Sep 07 11:22:51.794563 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.794538    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006ca005_7640_42ec_86c4_7a55ca8ff076.slice/crio-ea22e91f3cd79b041fb564ced9efc21e6a1bfa2162890bcef9ba40122b7e10df WatchSource:0}: Error finding container ea22e91f3cd79b041fb564ced9efc21e6a1bfa2162890bcef9ba40122b7e10df: Status 404 returned error can't find the container with id ea22e91f3cd79b041fb564ced9efc21e6a1bfa2162890bcef9ba40122b7e10df
Sep 07 11:22:51.795588 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.795567    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46731aa1_8800_4c5a_883a_563959a51a3c.slice/crio-b347b1ec4e639b75c4301868936df7ae679fb3688cb848528cc74fb44a6bc777 WatchSource:0}: Error finding container b347b1ec4e639b75c4301868936df7ae679fb3688cb848528cc74fb44a6bc777: Status 404 returned error can't find the container with id b347b1ec4e639b75c4301868936df7ae679fb3688cb848528cc74fb44a6bc777
Sep 07 11:22:51.817827 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.817800    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaabcd49_ae2b_4838_8cdf_4cf86ace72de.slice/crio-9909e933b88878cb57688ba932b4d91249c03e80ec098234a16d1c36d33b9af0 WatchSource:0}: Error finding container 9909e933b88878cb57688ba932b4d91249c03e80ec098234a16d1c36d33b9af0: Status 404 returned error can't find the container with id 9909e933b88878cb57688ba932b4d91249c03e80ec098234a16d1c36d33b9af0
Sep 07 11:22:51.819294 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.819163    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5c7178_3e5d_4ddc_aac9_e7406507d9bb.slice/crio-118c40cbc385c145208d453fca263d6842b8c2eeb99d5ecb0e621b6cc996400d WatchSource:0}: Error finding container 118c40cbc385c145208d453fca263d6842b8c2eeb99d5ecb0e621b6cc996400d: Status 404 returned error can't find the container with id 118c40cbc385c145208d453fca263d6842b8c2eeb99d5ecb0e621b6cc996400d
Sep 07 11:22:51.820506 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:22:51.820489    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008e2c25_f1aa_440f_9693_2e696410ef10.slice/crio-6d6b1613ae8563e3ad72a57a10847f502eecbf2fa88dfad4d14a8548f0aa857d WatchSource:0}: Error finding container 6d6b1613ae8563e3ad72a57a10847f502eecbf2fa88dfad4d14a8548f0aa857d: Status 404 returned error can't find the container with id 6d6b1613ae8563e3ad72a57a10847f502eecbf2fa88dfad4d14a8548f0aa857d
Sep 07 11:22:51.899561 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.899517    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 11:17:50 +0000 UTC" deadline="2028-03-12 05:04:10.305233612 +0000 UTC"
Sep 07 11:22:51.899561 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.899560    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13241h41m18.405676025s"
Sep 07 11:22:51.964083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.964054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerStarted","Data":"de498ac2cacf4e01a0c4295f855047ab31f11bdc9ffd1bcea0759229aef131ff"}
Sep 07 11:22:51.965250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.965220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-47js8" event={"ID":"32d447c3-1090-4e73-a7a4-c510811c26e3","Type":"ContainerStarted","Data":"8bcac4e48a300087fa7dd66d0cee9c3bca1c8d690d1709901b019e66cd78787b"}
Sep 07 11:22:51.967150 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.967126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" event={"ID":"07877c340319b68616068533f7d8229b","Type":"ContainerStarted","Data":"7721bb97b963c59030f63b3048b1bd8906693c471ec5c1a2562162f3bb5771e1"}
Sep 07 11:22:51.968402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.968379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-flng9" event={"ID":"008e2c25-f1aa-440f-9693-2e696410ef10","Type":"ContainerStarted","Data":"6d6b1613ae8563e3ad72a57a10847f502eecbf2fa88dfad4d14a8548f0aa857d"}
Sep 07 11:22:51.969598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.969573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ms8n9" event={"ID":"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb","Type":"ContainerStarted","Data":"118c40cbc385c145208d453fca263d6842b8c2eeb99d5ecb0e621b6cc996400d"}
Sep 07 11:22:51.970843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.970817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7j9h" event={"ID":"006ca005-7640-42ec-86c4-7a55ca8ff076","Type":"ContainerStarted","Data":"ea22e91f3cd79b041fb564ced9efc21e6a1bfa2162890bcef9ba40122b7e10df"}
Sep 07 11:22:51.971855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.971823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hkpwd" event={"ID":"50fe92fb-f492-45be-894d-7e4428d1985b","Type":"ContainerStarted","Data":"db45123b7eeead180f719d595bbe6229840dad954d4511a35169a04b5d45e326"}
Sep 07 11:22:51.973057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.973033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnkxf" event={"ID":"ff53ccaf-e144-41af-8248-489446634ebd","Type":"ContainerStarted","Data":"5f1c6daad32c08bcca4bbe75f8915531ee4af15209eb7ebcdb9760580201045d"}
Sep 07 11:22:51.974330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.974313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"9909e933b88878cb57688ba932b4d91249c03e80ec098234a16d1c36d33b9af0"}
Sep 07 11:22:51.975418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.975392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq" event={"ID":"46731aa1-8800-4c5a-883a-563959a51a3c","Type":"ContainerStarted","Data":"b347b1ec4e639b75c4301868936df7ae679fb3688cb848528cc74fb44a6bc777"}
Sep 07 11:22:51.986481 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:51.986423    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" podStartSLOduration=1.98640627 podStartE2EDuration="1.98640627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:51.985996321 +0000 UTC m=+3.460844495" watchObservedRunningTime="2026-09-07 11:22:51.98640627 +0000 UTC m=+3.461254422"
Sep 07 11:22:52.484819 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.484735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:52.485024 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.484821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:52.485024 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.484967    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:52.485024 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.485026    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret podName:3a1df1aa-6e57-4a72-8f6f-06c8588fab09 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:54.485007978 +0000 UTC m=+5.959856108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret") pod "global-pull-secret-syncer-sjsts" (UID: "3a1df1aa-6e57-4a72-8f6f-06c8588fab09") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:52.485416 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.485395    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:52.485552 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.485446    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:22:54.485432704 +0000 UTC m=+5.960280836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:52.686488 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.686440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:52.686668 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.686646    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:52.686736 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.686669    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:52.686736 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.686682    2578 projected.go:196] Error preparing data for projected volume kube-api-access-l7zlp for pod openshift-network-diagnostics/network-check-target-kjcnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:52.686835 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.686745    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp podName:91fce2a6-f2fd-4d41-9a30-e91037d9dc3b nodeName:}" failed. No retries permitted until 2026-09-07 11:22:54.686727773 +0000 UTC m=+6.161575903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7zlp" (UniqueName: "kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp") pod "network-check-target-kjcnz" (UID: "91fce2a6-f2fd-4d41-9a30-e91037d9dc3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:52.900100 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.900003    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 11:17:50 +0000 UTC" deadline="2028-04-21 17:07:08.54077915 +0000 UTC"
Sep 07 11:22:52.900100 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.900052    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14213h44m15.640731428s"
Sep 07 11:22:52.957518 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.956731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:52.957518 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.956871    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:22:52.957518 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.957122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:52.957518 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.957138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:52.957518 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.957244    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:22:52.957518 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:52.957323    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:22:52.997226 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.997192    2578 generic.go:356] "Generic (PLEG): container finished" podID="fda4d66429fdb419c55741b8bfee0a5b" containerID="2cce47ef338bacc9fed983c88e5ac1b00d95c5c907a9934c2710293a664bb569" exitCode=0
Sep 07 11:22:52.998121 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:52.998018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"fda4d66429fdb419c55741b8bfee0a5b","Type":"ContainerDied","Data":"2cce47ef338bacc9fed983c88e5ac1b00d95c5c907a9934c2710293a664bb569"}
Sep 07 11:22:54.018490 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:54.018005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"fda4d66429fdb419c55741b8bfee0a5b","Type":"ContainerStarted","Data":"ecf828bf8c26e42e72faad9c7fe711b7905ffe5ed90fca2d63ccc223b9c54f52"}
Sep 07 11:22:54.503025 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:54.502892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:54.503025 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:54.502987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:54.503256 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.503103    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:54.503256 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.503164    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:22:58.50314543 +0000 UTC m=+9.977993560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:54.503369 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.503317    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:54.503369 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.503367    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret podName:3a1df1aa-6e57-4a72-8f6f-06c8588fab09 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:58.503353921 +0000 UTC m=+9.978202056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret") pod "global-pull-secret-syncer-sjsts" (UID: "3a1df1aa-6e57-4a72-8f6f-06c8588fab09") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:54.705326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:54.705291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:54.705534 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.705482    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:54.705534 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.705503    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:54.705534 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.705515    2578 projected.go:196] Error preparing data for projected volume kube-api-access-l7zlp for pod openshift-network-diagnostics/network-check-target-kjcnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:54.705694 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.705573    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp podName:91fce2a6-f2fd-4d41-9a30-e91037d9dc3b nodeName:}" failed. No retries permitted until 2026-09-07 11:22:58.705554614 +0000 UTC m=+10.180402745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7zlp" (UniqueName: "kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp") pod "network-check-target-kjcnz" (UID: "91fce2a6-f2fd-4d41-9a30-e91037d9dc3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:54.954166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:54.954004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:54.954166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:54.954009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:54.954380 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.954261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:22:54.954380 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.954150    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:22:54.954380 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:54.954304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:54.954380 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:54.954375    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:22:56.954018 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:56.953968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:56.954018 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:56.954006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:56.954570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:56.953970    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:56.954570 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:56.954098    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:22:56.954570 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:56.954189    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:22:56.954570 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:56.954297    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:22:58.538760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:58.538486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:58.538760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:58.538577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:58.538760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.538646    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:58.538760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.538702    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:58.538760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.538707    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:23:06.538689416 +0000 UTC m=+18.013537560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:58.538760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.538736    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret podName:3a1df1aa-6e57-4a72-8f6f-06c8588fab09 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:06.538725725 +0000 UTC m=+18.013573852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret") pod "global-pull-secret-syncer-sjsts" (UID: "3a1df1aa-6e57-4a72-8f6f-06c8588fab09") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:58.739996 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:58.739957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:58.740213 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.740135    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:58.740213 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.740154    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:58.740213 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.740166    2578 projected.go:196] Error preparing data for projected volume kube-api-access-l7zlp for pod openshift-network-diagnostics/network-check-target-kjcnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:58.740377 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.740219    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp podName:91fce2a6-f2fd-4d41-9a30-e91037d9dc3b nodeName:}" failed. No retries permitted until 2026-09-07 11:23:06.740202652 +0000 UTC m=+18.215050782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7zlp" (UniqueName: "kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp") pod "network-check-target-kjcnz" (UID: "91fce2a6-f2fd-4d41-9a30-e91037d9dc3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:58.955217 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:58.955148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:22:58.955408 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.955245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:22:58.955408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:58.955308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:22:58.955408 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.955375    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:22:58.955782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:22:58.955737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:22:58.955862 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:22:58.955840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:00.954334 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:00.954084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:00.954805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:00.954084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:00.954805 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:00.954438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:00.954805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:00.954100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:00.954805 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:00.954544    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:00.954805 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:00.954637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:02.953799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:02.953762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:02.954214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:02.953770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:02.954214 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:02.953896    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:02.954214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:02.953907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:02.954214 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:02.953983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:02.954214 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:02.954103    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:04.953675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:04.953636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:04.954167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:04.953636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:04.954167 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:04.953784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:04.954167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:04.953636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:04.954167 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:04.953846    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:04.954167 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:04.953926    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:06.597259 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:06.597224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:06.597775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:06.597276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:06.597775 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.597377    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:23:06.597775 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.597395    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:23:06.597775 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.597455    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:23:22.597434224 +0000 UTC m=+34.072282363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:23:06.597775 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.597484    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret podName:3a1df1aa-6e57-4a72-8f6f-06c8588fab09 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:22.597477208 +0000 UTC m=+34.072325336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret") pod "global-pull-secret-syncer-sjsts" (UID: "3a1df1aa-6e57-4a72-8f6f-06c8588fab09") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:23:06.798767 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:06.798733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:06.798932 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.798913    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:23:06.798995 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.798937    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:23:06.798995 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.798948    2578 projected.go:196] Error preparing data for projected volume kube-api-access-l7zlp for pod openshift-network-diagnostics/network-check-target-kjcnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:23:06.799080 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.798998    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp podName:91fce2a6-f2fd-4d41-9a30-e91037d9dc3b nodeName:}" failed. No retries permitted until 2026-09-07 11:23:22.798985297 +0000 UTC m=+34.273833428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7zlp" (UniqueName: "kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp") pod "network-check-target-kjcnz" (UID: "91fce2a6-f2fd-4d41-9a30-e91037d9dc3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:23:06.954924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:06.954389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:06.954924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:06.954423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:06.954924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:06.954433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:06.954924 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.954541    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:06.954924 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.954617    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:06.954924 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:06.954702    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:08.954935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:08.954897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:08.955407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:08.955019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:08.955407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:08.955051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:08.955407 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:08.955130    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:08.955407 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:08.955015    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:08.955407 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:08.955224    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:10.053284 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.053136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-47js8" event={"ID":"32d447c3-1090-4e73-a7a4-c510811c26e3","Type":"ContainerStarted","Data":"a9cbbc3dc3b319850f2eae0129367f5aecb0b8f6dcfdf59a240c73ba89cb93f2"}
Sep 07 11:23:10.054651 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.054630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ms8n9" event={"ID":"bf5c7178-3e5d-4ddc-aac9-e7406507d9bb","Type":"ContainerStarted","Data":"872a0f85381c17ae49defb2df0019e6d9b71b8b4163a0914d7c649420fd13bbd"}
Sep 07 11:23:10.056058 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.056014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hkpwd" event={"ID":"50fe92fb-f492-45be-894d-7e4428d1985b","Type":"ContainerStarted","Data":"8587156e2b1a34094504ce5d7202ac3933b9c66b3bbb7639a96cdab01480a071"}
Sep 07 11:23:10.057512 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.057490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnkxf" event={"ID":"ff53ccaf-e144-41af-8248-489446634ebd","Type":"ContainerStarted","Data":"b680efc46988044a70843c6a08e78b9a66ee7efec162fb88937d8d9452e53518"}
Sep 07 11:23:10.059337 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.059323    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:23:10.059675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.059656    2578 generic.go:356] "Generic (PLEG): container finished" podID="aaabcd49-ae2b-4838-8cdf-4cf86ace72de" containerID="115ceb0155eeac71792de8b5539954a12640fd2634788ae6b603fc87a011532b" exitCode=1
Sep 07 11:23:10.059755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.059713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"89dcb749f3e9b7596996c117456ce28eaa578b8d18edbeef6102191b5e953c86"}
Sep 07 11:23:10.059802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.059766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerDied","Data":"115ceb0155eeac71792de8b5539954a12640fd2634788ae6b603fc87a011532b"}
Sep 07 11:23:10.059845 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.059810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"2727c587771f1ecdf32a17ca3db733bd0827219c3b8ad366ed1750547efda559"}
Sep 07 11:23:10.061287 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.061266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq" event={"ID":"46731aa1-8800-4c5a-883a-563959a51a3c","Type":"ContainerStarted","Data":"ae3dfbadb3e97a64de3b6a2454a61ad4c6b35adcaf3e79ce8b3282729f3e2573"}
Sep 07 11:23:10.062837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.062812    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dba6b53-96bc-4321-8260-caa270ad6ca9" containerID="2434b515a372c0f757db7a7258e377a391b2a0a8937d8a54c2680e6b433beb25" exitCode=0
Sep 07 11:23:10.062911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.062852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerDied","Data":"2434b515a372c0f757db7a7258e377a391b2a0a8937d8a54c2680e6b433beb25"}
Sep 07 11:23:10.072048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.072010    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" podStartSLOduration=20.071998978 podStartE2EDuration="20.071998978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:54.038157475 +0000 UTC m=+5.513005625" watchObservedRunningTime="2026-09-07 11:23:10.071998978 +0000 UTC m=+21.546847127"
Sep 07 11:23:10.072359 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.072321    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-47js8" podStartSLOduration=7.286724353 podStartE2EDuration="21.072309903s" totalImagesPullingTime="13.78558555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:10.071821167 +0000 UTC m=+21.546669316" watchObservedRunningTime="2026-09-07 11:23:10.072309903 +0000 UTC m=+21.547158057"
Sep 07 11:23:10.106176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.106138    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ms8n9" podStartSLOduration=3.326040683 podStartE2EDuration="21.106125388s" totalImagesPullingTime="17.780084705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:10.105700717 +0000 UTC m=+21.580548892" watchObservedRunningTime="2026-09-07 11:23:10.106125388 +0000 UTC m=+21.580973536"
Sep 07 11:23:10.122171 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.122133    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hkpwd" podStartSLOduration=3.337925299 podStartE2EDuration="21.122118571s" totalImagesPullingTime="17.784193272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:10.121993123 +0000 UTC m=+21.596841293" watchObservedRunningTime="2026-09-07 11:23:10.122118571 +0000 UTC m=+21.596966719"
Sep 07 11:23:10.135348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.135310    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vnkxf" podStartSLOduration=3.325617971 podStartE2EDuration="21.135296233s" totalImagesPullingTime="17.809678262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:10.134984846 +0000 UTC m=+21.609833006" watchObservedRunningTime="2026-09-07 11:23:10.135296233 +0000 UTC m=+21.610144382"
Sep 07 11:23:10.248588 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.248485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:10.314216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.314182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:10.738656 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.738618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:10.839198 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.839160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:10.939765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.939691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:10.953618 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.953592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:10.953742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.953605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:10.953742 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:10.953718    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:10.953742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:10.953739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:10.953884 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:10.953853    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:10.953964 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:10.953943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:11.041230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.041200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:11.066993 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.066962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-flng9" event={"ID":"008e2c25-f1aa-440f-9693-2e696410ef10","Type":"ContainerStarted","Data":"dc4c54169c1346ad2ff68dd8e831e369ce4e3bdb853d566575af875e2acdc82c"}
Sep 07 11:23:11.069576 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.069552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7j9h" event={"ID":"006ca005-7640-42ec-86c4-7a55ca8ff076","Type":"ContainerStarted","Data":"2e4ee72c97d34377ee15210e874e643cc4884d0f0ed818170b32fe4f0f94343c"}
Sep 07 11:23:11.072851 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.072829    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:23:11.073290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.073267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"4b34145fc88aa790f91b99c4a214f0e7252a273fa6b29745221d551cf434a374"}
Sep 07 11:23:11.073400 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.073326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"2ff495f7ad3479538a0a604cac288aefb9cd32ba1777e8c14d8a3c5116a487ae"}
Sep 07 11:23:11.073400 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.073357    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"829c96ddc58e61f92c8b9edb590f850ecd625003011fd5ae64b59aeef81f030c"}
Sep 07 11:23:11.087297 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.087246    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-flng9" podStartSLOduration=4.309460315 podStartE2EDuration="22.087228563s" totalImagesPullingTime="17.777768248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:11.086431581 +0000 UTC m=+22.561279731" watchObservedRunningTime="2026-09-07 11:23:11.087228563 +0000 UTC m=+22.562076712"
Sep 07 11:23:11.107630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.107581    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k7j9h" podStartSLOduration=3.847928602 podStartE2EDuration="22.107562302s" totalImagesPullingTime="18.2596337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:11.106880618 +0000 UTC m=+22.581728793" watchObservedRunningTime="2026-09-07 11:23:11.107562302 +0000 UTC m=+22.582410454"
Sep 07 11:23:11.144508 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.144484    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 11:23:11.712452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:11.712408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:12.005151 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.005064    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:12.064015 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.063910    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T11:23:11.144506004Z","UUID":"89ee2eb6-3258-4a6b-8a5d-8d33de27d717","Handler":null,"Name":"","Endpoint":""}
Sep 07 11:23:12.065304 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.065270    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 11:23:12.065304 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.065304    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 11:23:12.077418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.077386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq" event={"ID":"46731aa1-8800-4c5a-883a-563959a51a3c","Type":"ContainerStarted","Data":"80872b3501ef20e7e49e46956ffdb1693e8dab0f3569e3ef7208b32e3122eafb"}
Sep 07 11:23:12.117766 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.117728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:12.203199 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.203168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:12.897126 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.897092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:12.954560 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.954490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:12.954560 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.954515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:12.954782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.954490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:12.954782 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:12.954611    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:12.954782 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:12.954672    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:12.954936 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:12.954789    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:12.996933 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:12.996901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:13.082122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:13.082081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq" event={"ID":"46731aa1-8800-4c5a-883a-563959a51a3c","Type":"ContainerStarted","Data":"201dedaf3234ee89bd40b4c2d48b50c9961ff46d1c2b21b9081caa82aa80974a"}
Sep 07 11:23:13.085829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:13.085801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:23:13.086245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:13.086193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"bbf63a268f1d28bbca23dc2637c9f28d9dafe4347481b67d905d96ef426a8ec7"}
Sep 07 11:23:13.101587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:13.101542    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnfrq" podStartSLOduration=4.060819554 podStartE2EDuration="24.101529117s" totalImagesPullingTime="20.040709563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:13.101131182 +0000 UTC m=+24.575979350" watchObservedRunningTime="2026-09-07 11:23:13.101529117 +0000 UTC m=+24.576377270"
Sep 07 11:23:13.317914 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:13.317838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:13.387493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:13.387447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:14.181426 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:14.181396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:23:14.182027 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:14.182011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:23:14.565927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:14.565896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:14.954436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:14.954411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:14.954619 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:14.954532    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:14.954619 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:14.954417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:14.954740 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:14.954671    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:14.954740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:14.954708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:14.954834 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:14.954784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:15.093057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.093032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:23:15.093439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.093418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"bd17c2432308fa7921b11401de5a639f24f4f0babb0fcc702f5481c84af4902e"}
Sep 07 11:23:15.093675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.093652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:23:15.093880 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.093862    2578 scope.go:117] "RemoveContainer" containerID="115ceb0155eeac71792de8b5539954a12640fd2634788ae6b603fc87a011532b"
Sep 07 11:23:15.095747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.095723    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dba6b53-96bc-4321-8260-caa270ad6ca9" containerID="38dd130efc37e5a7ee376382cdf74036d1724248ed9eb169ea395001a6ad5930" exitCode=0
Sep 07 11:23:15.095854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.095802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerDied","Data":"38dd130efc37e5a7ee376382cdf74036d1724248ed9eb169ea395001a6ad5930"}
Sep 07 11:23:15.096170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.096119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:23:15.096541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.096448    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-47js8"
Sep 07 11:23:15.110332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.110313    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:23:15.215699 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:15.215632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:16.102586 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.102559    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:23:16.102970 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.102939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" event={"ID":"aaabcd49-ae2b-4838-8cdf-4cf86ace72de","Type":"ContainerStarted","Data":"9d32ddd32c5c794727ff8fc2ff9365ba7bab528845dd9e5c016d0a693017fcfa"}
Sep 07 11:23:16.103448 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.103248    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:23:16.103448 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.103339    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:23:16.117896 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.117875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:23:16.134335 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.134287    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb" podStartSLOduration=9.333121261 podStartE2EDuration="27.134266492s" totalImagesPullingTime="17.801145231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:16.132666773 +0000 UTC m=+27.607514944" watchObservedRunningTime="2026-09-07 11:23:16.134266492 +0000 UTC m=+27.609114642"
Sep 07 11:23:16.419796 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.419697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sjsts"]
Sep 07 11:23:16.420285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.419841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:16.420285 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:16.419940    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:16.424456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.424404    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjcnz"]
Sep 07 11:23:16.424988 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.424960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:16.425096 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:16.425068    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:16.425377 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.425354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lthxp"]
Sep 07 11:23:16.425517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:16.425457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:16.425626 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:16.425606    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:17.106597 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:17.106561    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dba6b53-96bc-4321-8260-caa270ad6ca9" containerID="59141ae0974c59e137497ee8616e8cd7f55beb4d552be69f0d1f0dcd91e4c585" exitCode=0
Sep 07 11:23:17.106759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:17.106650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerDied","Data":"59141ae0974c59e137497ee8616e8cd7f55beb4d552be69f0d1f0dcd91e4c585"}
Sep 07 11:23:17.954308 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:17.954277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:17.954760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:17.954404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:17.954760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:17.954414    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:17.954760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:17.954500    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:17.954760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:17.954517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:17.954760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:17.954615    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:19.117162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:19.117127    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dba6b53-96bc-4321-8260-caa270ad6ca9" containerID="dc8d5ff0e5160a2aecb6c7fdcfa950e2888a1c48fd2e741e25aa2b0e7a68e0ac" exitCode=0
Sep 07 11:23:19.117548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:19.117204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerDied","Data":"dc8d5ff0e5160a2aecb6c7fdcfa950e2888a1c48fd2e741e25aa2b0e7a68e0ac"}
Sep 07 11:23:19.953801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:19.953769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:19.953984 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:19.953769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:19.953984 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:19.953885    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjsts" podUID="3a1df1aa-6e57-4a72-8f6f-06c8588fab09"
Sep 07 11:23:19.953984 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:19.953974    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjcnz" podUID="91fce2a6-f2fd-4d41-9a30-e91037d9dc3b"
Sep 07 11:23:19.954159 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:19.953777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:19.954159 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:19.954107    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lthxp" podUID="a90bddad-de76-4653-8a1b-e3546fb5aa7f"
Sep 07 11:23:21.805007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.804921    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeReady"
Sep 07 11:23:21.805420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.805072    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 11:23:21.850661 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.850627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tv695"]
Sep 07 11:23:21.854739 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.854704    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2tfrr"]
Sep 07 11:23:21.854872 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.854858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:21.857322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.857290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ph89j\""
Sep 07 11:23:21.857624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.857607    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 11:23:21.857871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.857847    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 11:23:21.858348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.858327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:21.860628 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.860571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 11:23:21.860746 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.860710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4j7np\""
Sep 07 11:23:21.860800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.860768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 11:23:21.860895 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.860873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 11:23:21.861137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.861086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2tfrr"]
Sep 07 11:23:21.864572 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.863711    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tv695"]
Sep 07 11:23:21.950697 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.950661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:21.953561 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.953535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:21.953684 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.953542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:21.953746 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.953548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:21.956354 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.956304    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8r9c\""
Sep 07 11:23:21.956354 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.956323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 11:23:21.956574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.956323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 11:23:21.956574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.956430    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 11:23:21.956668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.956588    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qlhnv\""
Sep 07 11:23:21.956668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:21.956601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 11:23:22.006316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.006287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879x9\" (UniqueName: \"kubernetes.io/projected/21838826-4304-4ee2-a828-5e201e7ff2d9-kube-api-access-879x9\") pod \"ingress-canary-2tfrr\" (UID: \"21838826-4304-4ee2-a828-5e201e7ff2d9\") " pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:22.006439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.006378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52727a38-dbef-437a-96ec-4401f1a6292e-config-volume\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.006439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.006431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21838826-4304-4ee2-a828-5e201e7ff2d9-cert\") pod \"ingress-canary-2tfrr\" (UID: \"21838826-4304-4ee2-a828-5e201e7ff2d9\") " pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:22.006567 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.006483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52727a38-dbef-437a-96ec-4401f1a6292e-tmp-dir\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.006604 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.006574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.006652 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.006595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pzp\" (UniqueName: \"kubernetes.io/projected/52727a38-dbef-437a-96ec-4401f1a6292e-kube-api-access-w8pzp\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.107801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.107727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21838826-4304-4ee2-a828-5e201e7ff2d9-cert\") pod \"ingress-canary-2tfrr\" (UID: \"21838826-4304-4ee2-a828-5e201e7ff2d9\") " pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:22.107801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.107773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52727a38-dbef-437a-96ec-4401f1a6292e-tmp-dir\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.108038 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.108008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.108133 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.108049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pzp\" (UniqueName: \"kubernetes.io/projected/52727a38-dbef-437a-96ec-4401f1a6292e-kube-api-access-w8pzp\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.108133 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.108090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52727a38-dbef-437a-96ec-4401f1a6292e-tmp-dir\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.108133 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.108102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879x9\" (UniqueName: \"kubernetes.io/projected/21838826-4304-4ee2-a828-5e201e7ff2d9-kube-api-access-879x9\") pod \"ingress-canary-2tfrr\" (UID: \"21838826-4304-4ee2-a828-5e201e7ff2d9\") " pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:22.108133 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:22.108120    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:22.108328 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.108157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52727a38-dbef-437a-96ec-4401f1a6292e-config-volume\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.108328 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:22.108194    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls podName:52727a38-dbef-437a-96ec-4401f1a6292e nodeName:}" failed. No retries permitted until 2026-09-07 11:23:22.608171512 +0000 UTC m=+34.083019660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls") pod "dns-default-tv695" (UID: "52727a38-dbef-437a-96ec-4401f1a6292e") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:22.108717 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.108691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52727a38-dbef-437a-96ec-4401f1a6292e-config-volume\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.112436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.112410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21838826-4304-4ee2-a828-5e201e7ff2d9-cert\") pod \"ingress-canary-2tfrr\" (UID: \"21838826-4304-4ee2-a828-5e201e7ff2d9\") " pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:22.117121 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.116971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pzp\" (UniqueName: \"kubernetes.io/projected/52727a38-dbef-437a-96ec-4401f1a6292e-kube-api-access-w8pzp\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.117121 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.117116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-879x9\" (UniqueName: \"kubernetes.io/projected/21838826-4304-4ee2-a828-5e201e7ff2d9-kube-api-access-879x9\") pod \"ingress-canary-2tfrr\" (UID: \"21838826-4304-4ee2-a828-5e201e7ff2d9\") " pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:22.178776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.178747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2tfrr"
Sep 07 11:23:22.611913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.611875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:22.612101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.611929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:22.612101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.611958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:22.612101 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:22.612031    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 11:23:22.612101 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:22.612068    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:22.612309 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:22.612115    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:23:54.612095274 +0000 UTC m=+66.086943417 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : secret "metrics-daemon-secret" not found
Sep 07 11:23:22.612309 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:22.612129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls podName:52727a38-dbef-437a-96ec-4401f1a6292e nodeName:}" failed. No retries permitted until 2026-09-07 11:23:23.612123431 +0000 UTC m=+35.086971558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls") pod "dns-default-tv695" (UID: "52727a38-dbef-437a-96ec-4401f1a6292e") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:22.614603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.614578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1df1aa-6e57-4a72-8f6f-06c8588fab09-original-pull-secret\") pod \"global-pull-secret-syncer-sjsts\" (UID: \"3a1df1aa-6e57-4a72-8f6f-06c8588fab09\") " pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:22.813325 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.813234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:22.816173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.816147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7zlp\" (UniqueName: \"kubernetes.io/projected/91fce2a6-f2fd-4d41-9a30-e91037d9dc3b-kube-api-access-l7zlp\") pod \"network-check-target-kjcnz\" (UID: \"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b\") " pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:22.865043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.865015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjsts"
Sep 07 11:23:22.871611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:22.871591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:23.062565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:23.062531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:23.294353 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:23.294318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2tfrr"]
Sep 07 11:23:23.297319 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:23.297284    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sjsts"]
Sep 07 11:23:23.298098 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:23.298076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjcnz"]
Sep 07 11:23:23.306659 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:23:23.306630    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21838826_4304_4ee2_a828_5e201e7ff2d9.slice/crio-9703e1244508d8699dce474fecfc03d3dcce9a284328602a8caaf1d232f63016 WatchSource:0}: Error finding container 9703e1244508d8699dce474fecfc03d3dcce9a284328602a8caaf1d232f63016: Status 404 returned error can't find the container with id 9703e1244508d8699dce474fecfc03d3dcce9a284328602a8caaf1d232f63016
Sep 07 11:23:23.306916 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:23:23.306895    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fce2a6_f2fd_4d41_9a30_e91037d9dc3b.slice/crio-a842468cba8317f0293125cfa470454fb66afb6662142122e85a4e27aa53863c WatchSource:0}: Error finding container a842468cba8317f0293125cfa470454fb66afb6662142122e85a4e27aa53863c: Status 404 returned error can't find the container with id a842468cba8317f0293125cfa470454fb66afb6662142122e85a4e27aa53863c
Sep 07 11:23:23.307111 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:23:23.307088    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1df1aa_6e57_4a72_8f6f_06c8588fab09.slice/crio-19f5a43635e8ace4a7014e545a96c6c88749ee9b5ea4f75862680291f4c0f645 WatchSource:0}: Error finding container 19f5a43635e8ace4a7014e545a96c6c88749ee9b5ea4f75862680291f4c0f645: Status 404 returned error can't find the container with id 19f5a43635e8ace4a7014e545a96c6c88749ee9b5ea4f75862680291f4c0f645
Sep 07 11:23:23.620176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:23.620146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:23.620344 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:23.620271    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:23.620344 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:23.620338    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls podName:52727a38-dbef-437a-96ec-4401f1a6292e nodeName:}" failed. No retries permitted until 2026-09-07 11:23:25.620319404 +0000 UTC m=+37.095167548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls") pod "dns-default-tv695" (UID: "52727a38-dbef-437a-96ec-4401f1a6292e") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:24.130544 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:24.130498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2tfrr" event={"ID":"21838826-4304-4ee2-a828-5e201e7ff2d9","Type":"ContainerStarted","Data":"9703e1244508d8699dce474fecfc03d3dcce9a284328602a8caaf1d232f63016"}
Sep 07 11:23:24.132423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:24.132367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjcnz" event={"ID":"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b","Type":"ContainerStarted","Data":"a842468cba8317f0293125cfa470454fb66afb6662142122e85a4e27aa53863c"}
Sep 07 11:23:24.136499 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:24.136451    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dba6b53-96bc-4321-8260-caa270ad6ca9" containerID="cd5bbccdd464df8a0e8d26b56eb03e9f22803767ac4eea529bd98a1fabd74d2b" exitCode=0
Sep 07 11:23:24.136607 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:24.136554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerDied","Data":"cd5bbccdd464df8a0e8d26b56eb03e9f22803767ac4eea529bd98a1fabd74d2b"}
Sep 07 11:23:24.139041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:24.139001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sjsts" event={"ID":"3a1df1aa-6e57-4a72-8f6f-06c8588fab09","Type":"ContainerStarted","Data":"19f5a43635e8ace4a7014e545a96c6c88749ee9b5ea4f75862680291f4c0f645"}
Sep 07 11:23:25.148429 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:25.148312    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dba6b53-96bc-4321-8260-caa270ad6ca9" containerID="4a472121f3d2db5b32a8b5167674b6fb98990cf38ca21381a5e400790f30c65b" exitCode=0
Sep 07 11:23:25.148429 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:25.148377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerDied","Data":"4a472121f3d2db5b32a8b5167674b6fb98990cf38ca21381a5e400790f30c65b"}
Sep 07 11:23:25.636114 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:25.635866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:25.636282 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:25.636017    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:25.636282 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:25.636264    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls podName:52727a38-dbef-437a-96ec-4401f1a6292e nodeName:}" failed. No retries permitted until 2026-09-07 11:23:29.636242262 +0000 UTC m=+41.111090390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls") pod "dns-default-tv695" (UID: "52727a38-dbef-437a-96ec-4401f1a6292e") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:25.832156 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:25.832093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:25.901165 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:25.901092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:26.155728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:26.155619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" event={"ID":"4dba6b53-96bc-4321-8260-caa270ad6ca9","Type":"ContainerStarted","Data":"8a47eacdb23c0871ba19f2e8c5310a7413cf4a6e844b06e6233c69db9a29a623"}
Sep 07 11:23:26.178368 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:26.178313    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bwz6m" podStartSLOduration=9.1590284 podStartE2EDuration="37.178293322s" totalImagesPullingTime="28.019264922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:26.177144427 +0000 UTC m=+37.651992601" watchObservedRunningTime="2026-09-07 11:23:26.178293322 +0000 UTC m=+37.653141463"
Sep 07 11:23:26.298424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:26.298389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:26.398389 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:26.398354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:26.465157 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:26.465121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:28.212385 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:28.212343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:28.274951 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:28.274916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:29.165394 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.165347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sjsts" event={"ID":"3a1df1aa-6e57-4a72-8f6f-06c8588fab09","Type":"ContainerStarted","Data":"83b02133538e0c734fdda71a46635d1ba0617fa843c2b0b473dc87797bf01cfd"}
Sep 07 11:23:29.167224 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.167194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2tfrr" event={"ID":"21838826-4304-4ee2-a828-5e201e7ff2d9","Type":"ContainerStarted","Data":"8938311d66890132d880afeb3d2e9dc7d0ccf8da7d406bad104a1cca86e91c80"}
Sep 07 11:23:29.169045 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.169022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjcnz" event={"ID":"91fce2a6-f2fd-4d41-9a30-e91037d9dc3b","Type":"ContainerStarted","Data":"bd57e870aeac6a545f1461d10226350e1b67d3937600c7ba05f8c66d28ce325c"}
Sep 07 11:23:29.169185 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.169166    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:23:29.184850 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.184797    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sjsts" podStartSLOduration=34.565546831 podStartE2EDuration="40.184776467s" totalImagesPullingTime="5.619229636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:29.184639881 +0000 UTC m=+40.659488065" watchObservedRunningTime="2026-09-07 11:23:29.184776467 +0000 UTC m=+40.659624615"
Sep 07 11:23:29.201910 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.201852    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kjcnz" podStartSLOduration=34.594615538 podStartE2EDuration="40.201833275s" totalImagesPullingTime="5.607217737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:29.201597134 +0000 UTC m=+40.676445305" watchObservedRunningTime="2026-09-07 11:23:29.201833275 +0000 UTC m=+40.676681418"
Sep 07 11:23:29.216897 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.216851    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2tfrr" podStartSLOduration=2.629504796 podStartE2EDuration="8.216837728s" totalImagesPullingTime="5.587332932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:29.216248463 +0000 UTC m=+40.691096613" watchObservedRunningTime="2026-09-07 11:23:29.216837728 +0000 UTC m=+40.691685904"
Sep 07 11:23:29.665187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:29.665143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:29.665404 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:29.665304    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:29.665404 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:29.665372    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls podName:52727a38-dbef-437a-96ec-4401f1a6292e nodeName:}" failed. No retries permitted until 2026-09-07 11:23:37.665355355 +0000 UTC m=+49.140203482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls") pod "dns-default-tv695" (UID: "52727a38-dbef-437a-96ec-4401f1a6292e") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:31.257205 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:31.257175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:31.385643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:31.385606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:37.719887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:37.719846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:37.720274 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:37.720022    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:37.720274 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:37.720101    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls podName:52727a38-dbef-437a-96ec-4401f1a6292e nodeName:}" failed. No retries permitted until 2026-09-07 11:23:53.720084052 +0000 UTC m=+65.194932179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls") pod "dns-default-tv695" (UID: "52727a38-dbef-437a-96ec-4401f1a6292e") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:47.822089 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:47.822057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:47.882757 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:47.882731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:48.127621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:48.127596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-br8kb"
Sep 07 11:23:48.250294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:48.250259    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:48.316145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:48.316113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:53.728852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:53.728807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:23:53.729315 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:53.728927    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:53.729315 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:53.728982    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls podName:52727a38-dbef-437a-96ec-4401f1a6292e nodeName:}" failed. No retries permitted until 2026-09-07 11:24:25.728969034 +0000 UTC m=+97.203817160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls") pod "dns-default-tv695" (UID: "52727a38-dbef-437a-96ec-4401f1a6292e") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:54.634419 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:23:54.634372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:23:54.634617 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:54.634542    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 11:23:54.634617 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:23:54.634600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs podName:a90bddad-de76-4653-8a1b-e3546fb5aa7f nodeName:}" failed. No retries permitted until 2026-09-07 11:24:58.634586311 +0000 UTC m=+130.109434437 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs") pod "network-metrics-daemon-lthxp" (UID: "a90bddad-de76-4653-8a1b-e3546fb5aa7f") : secret "metrics-daemon-secret" not found
Sep 07 11:24:00.176726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:00.176694    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kjcnz"
Sep 07 11:24:01.964170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.964136    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-vhznl"]
Sep 07 11:24:01.967213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.967195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl"
Sep 07 11:24:01.969634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.969601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vxlnv\""
Sep 07 11:24:01.983480 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.983434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-vhznl"]
Sep 07 11:24:01.984345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.984317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkf5d\" (UniqueName: \"kubernetes.io/projected/61f0563e-6e58-46dc-be37-2e67461139c4-kube-api-access-mkf5d\") pod \"network-check-source-685456655f-vhznl\" (UID: \"61f0563e-6e58-46dc-be37-2e67461139c4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl"
Sep 07 11:24:01.989706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.989684    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"]
Sep 07 11:24:01.991593 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.991578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:01.994173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.994123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 07 11:24:01.994173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.994169    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 07 11:24:01.994318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.994210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 07 11:24:01.994318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.994226    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:24:01.994452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:01.994341    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g9rzx\""
Sep 07 11:24:02.003176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.003152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"]
Sep 07 11:24:02.069912 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.069876    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"]
Sep 07 11:24:02.071952 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.071935    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5xm2q"]
Sep 07 11:24:02.072075 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.072055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.073665 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.073644    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z5hz6"]
Sep 07 11:24:02.073798 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.073762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.074671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.074642    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-smzsr\""
Sep 07 11:24:02.074671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.074655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 11:24:02.074671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.074652    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 07 11:24:02.074865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.074689    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 11:24:02.075536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.075519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.076476 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.076438    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 07 11:24:02.076827 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.076813    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 07 11:24:02.077540 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.077500    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 07 11:24:02.077653 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.077559    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:24:02.077653 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.077591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 07 11:24:02.077653 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.077638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fdgsp\""
Sep 07 11:24:02.077972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.077952    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 11:24:02.078067 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.078041    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 11:24:02.078601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.078577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 07 11:24:02.078691 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.078618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 07 11:24:02.078752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.078621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bhbzf\""
Sep 07 11:24:02.083627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.083606    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 07 11:24:02.084718 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.084698    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 07 11:24:02.085341 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmd66\" (UniqueName: \"kubernetes.io/projected/fc1f1428-c3e4-4865-8bf8-e713111191a2-kube-api-access-qmd66\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.085438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3b1517-0672-4bff-9738-3bc5682ea071-serving-cert\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.085438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkf5d\" (UniqueName: \"kubernetes.io/projected/61f0563e-6e58-46dc-be37-2e67461139c4-kube-api-access-mkf5d\") pod \"network-check-source-685456655f-vhznl\" (UID: \"61f0563e-6e58-46dc-be37-2e67461139c4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl"
Sep 07 11:24:02.085438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4c3b1517-0672-4bff-9738-3bc5682ea071-snapshots\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.085438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7954bc9a-b5a5-4829-aed7-d4004433d11c-serving-cert\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.085674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7954bc9a-b5a5-4829-aed7-d4004433d11c-trusted-ca\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.085674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.085674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmpt\" (UniqueName: \"kubernetes.io/projected/4c3b1517-0672-4bff-9738-3bc5682ea071-kube-api-access-xfmpt\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.085674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fc1f1428-c3e4-4865-8bf8-e713111191a2-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.085674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7954bc9a-b5a5-4829-aed7-d4004433d11c-config\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.085674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a872110-863f-486c-b6d6-ff742fa69a5f-serving-cert\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.085674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a872110-863f-486c-b6d6-ff742fa69a5f-config\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.086020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c3b1517-0672-4bff-9738-3bc5682ea071-tmp\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.086020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3b1517-0672-4bff-9738-3bc5682ea071-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.086020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpf5d\" (UniqueName: \"kubernetes.io/projected/7a872110-863f-486c-b6d6-ff742fa69a5f-kube-api-access-gpf5d\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.086020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3b1517-0672-4bff-9738-3bc5682ea071-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.086020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.085788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcrqp\" (UniqueName: \"kubernetes.io/projected/7954bc9a-b5a5-4829-aed7-d4004433d11c-kube-api-access-xcrqp\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.089991 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.089970    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5xm2q"]
Sep 07 11:24:02.090712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.090690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z5hz6"]
Sep 07 11:24:02.092078 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.092056    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"]
Sep 07 11:24:02.100831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.100809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkf5d\" (UniqueName: \"kubernetes.io/projected/61f0563e-6e58-46dc-be37-2e67461139c4-kube-api-access-mkf5d\") pod \"network-check-source-685456655f-vhznl\" (UID: \"61f0563e-6e58-46dc-be37-2e67461139c4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl"
Sep 07 11:24:02.141593 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.141556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:02.162997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.162964    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"]
Sep 07 11:24:02.164999 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.164982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.167430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.167409    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 07 11:24:02.167602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.167507    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 07 11:24:02.168046 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.168019    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:24:02.168135 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.168121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gxnw5\""
Sep 07 11:24:02.169930 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.169906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 07 11:24:02.174209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.174185    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"]
Sep 07 11:24:02.186677 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4c3b1517-0672-4bff-9738-3bc5682ea071-snapshots\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.186677 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7954bc9a-b5a5-4829-aed7-d4004433d11c-serving-cert\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.186839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7954bc9a-b5a5-4829-aed7-d4004433d11c-trusted-ca\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.186839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.186839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/383ede9c-a994-4f54-88f9-c4647a3e6c8d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.186839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmpt\" (UniqueName: \"kubernetes.io/projected/4c3b1517-0672-4bff-9738-3bc5682ea071-kube-api-access-xfmpt\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.186839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/383ede9c-a994-4f54-88f9-c4647a3e6c8d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.187082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fc1f1428-c3e4-4865-8bf8-e713111191a2-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.187082 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:02.186914    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:02.187082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7954bc9a-b5a5-4829-aed7-d4004433d11c-config\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.187082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.186962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a872110-863f-486c-b6d6-ff742fa69a5f-serving-cert\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.187082 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:02.187000    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls podName:fc1f1428-c3e4-4865-8bf8-e713111191a2 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:02.686977823 +0000 UTC m=+74.161825971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-v725n" (UID: "fc1f1428-c3e4-4865-8bf8-e713111191a2") : secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:02.187082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a872110-863f-486c-b6d6-ff742fa69a5f-config\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.187082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c3b1517-0672-4bff-9738-3bc5682ea071-tmp\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.187483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3b1517-0672-4bff-9738-3bc5682ea071-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.187483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4c3b1517-0672-4bff-9738-3bc5682ea071-snapshots\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.187652 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fc1f1428-c3e4-4865-8bf8-e713111191a2-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.187724 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a872110-863f-486c-b6d6-ff742fa69a5f-config\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.187724 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7954bc9a-b5a5-4829-aed7-d4004433d11c-trusted-ca\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.187822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpf5d\" (UniqueName: \"kubernetes.io/projected/7a872110-863f-486c-b6d6-ff742fa69a5f-kube-api-access-gpf5d\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.187822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3b1517-0672-4bff-9738-3bc5682ea071-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.187822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcrqp\" (UniqueName: \"kubernetes.io/projected/7954bc9a-b5a5-4829-aed7-d4004433d11c-kube-api-access-xcrqp\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.187977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lwn\" (UniqueName: \"kubernetes.io/projected/383ede9c-a994-4f54-88f9-c4647a3e6c8d-kube-api-access-j9lwn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.187977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c3b1517-0672-4bff-9738-3bc5682ea071-tmp\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.187977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmd66\" (UniqueName: \"kubernetes.io/projected/fc1f1428-c3e4-4865-8bf8-e713111191a2-kube-api-access-qmd66\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.187977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.187939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3b1517-0672-4bff-9738-3bc5682ea071-serving-cert\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.188192 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.188076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3b1517-0672-4bff-9738-3bc5682ea071-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.188246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.188190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7954bc9a-b5a5-4829-aed7-d4004433d11c-config\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.188528 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.188500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3b1517-0672-4bff-9738-3bc5682ea071-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.189417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.189394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a872110-863f-486c-b6d6-ff742fa69a5f-serving-cert\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.189531 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.189425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7954bc9a-b5a5-4829-aed7-d4004433d11c-serving-cert\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.190006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.189990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3b1517-0672-4bff-9738-3bc5682ea071-serving-cert\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.198664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.198641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpf5d\" (UniqueName: \"kubernetes.io/projected/7a872110-863f-486c-b6d6-ff742fa69a5f-kube-api-access-gpf5d\") pod \"service-ca-operator-859f94c4b-cqztt\" (UID: \"7a872110-863f-486c-b6d6-ff742fa69a5f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.198835 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.198816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmd66\" (UniqueName: \"kubernetes.io/projected/fc1f1428-c3e4-4865-8bf8-e713111191a2-kube-api-access-qmd66\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.198924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.198907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcrqp\" (UniqueName: \"kubernetes.io/projected/7954bc9a-b5a5-4829-aed7-d4004433d11c-kube-api-access-xcrqp\") pod \"console-operator-85b454874b-5xm2q\" (UID: \"7954bc9a-b5a5-4829-aed7-d4004433d11c\") " pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.199242 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.199224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmpt\" (UniqueName: \"kubernetes.io/projected/4c3b1517-0672-4bff-9738-3bc5682ea071-kube-api-access-xfmpt\") pod \"insights-operator-76bcd6ccf5-z5hz6\" (UID: \"4c3b1517-0672-4bff-9738-3bc5682ea071\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.275916 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.275825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl"
Sep 07 11:24:02.288726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.288689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lwn\" (UniqueName: \"kubernetes.io/projected/383ede9c-a994-4f54-88f9-c4647a3e6c8d-kube-api-access-j9lwn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.288879 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.288782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/383ede9c-a994-4f54-88f9-c4647a3e6c8d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.288879 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.288811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/383ede9c-a994-4f54-88f9-c4647a3e6c8d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.289395 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.289364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/383ede9c-a994-4f54-88f9-c4647a3e6c8d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.291055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.291031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/383ede9c-a994-4f54-88f9-c4647a3e6c8d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.297081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.297055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lwn\" (UniqueName: \"kubernetes.io/projected/383ede9c-a994-4f54-88f9-c4647a3e6c8d-kube-api-access-j9lwn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fpjjx\" (UID: \"383ede9c-a994-4f54-88f9-c4647a3e6c8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.299943 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.299919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"
Sep 07 11:24:02.378439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.378408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:02.389512 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.389486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:02.396444 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.396422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6"
Sep 07 11:24:02.475811 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.475777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"
Sep 07 11:24:02.548498 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.548386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-vhznl"]
Sep 07 11:24:02.551055 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:02.551030    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f0563e_6e58_46dc_be37_2e67461139c4.slice/crio-de0fa0ae04f84a435f4c3b0d3124e75b0914be6059994946e4e4e36745c90744 WatchSource:0}: Error finding container de0fa0ae04f84a435f4c3b0d3124e75b0914be6059994946e4e4e36745c90744: Status 404 returned error can't find the container with id de0fa0ae04f84a435f4c3b0d3124e75b0914be6059994946e4e4e36745c90744
Sep 07 11:24:02.576540 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.576508    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt"]
Sep 07 11:24:02.579322 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:02.579295    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a872110_863f_486c_b6d6_ff742fa69a5f.slice/crio-1ef3ad7b3ebf937fce26464c7cee38c5bc0a1ef519f79ae43563179ae4031a59 WatchSource:0}: Error finding container 1ef3ad7b3ebf937fce26464c7cee38c5bc0a1ef519f79ae43563179ae4031a59: Status 404 returned error can't find the container with id 1ef3ad7b3ebf937fce26464c7cee38c5bc0a1ef519f79ae43563179ae4031a59
Sep 07 11:24:02.659299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.659267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5xm2q"]
Sep 07 11:24:02.662249 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:02.662220    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7954bc9a_b5a5_4829_aed7_d4004433d11c.slice/crio-545804e181c5e0b803753dd4f1f7f5bd97bfe7888880a6e587c939466dd55437 WatchSource:0}: Error finding container 545804e181c5e0b803753dd4f1f7f5bd97bfe7888880a6e587c939466dd55437: Status 404 returned error can't find the container with id 545804e181c5e0b803753dd4f1f7f5bd97bfe7888880a6e587c939466dd55437
Sep 07 11:24:02.679964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.679940    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z5hz6"]
Sep 07 11:24:02.682931 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:02.682901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3b1517_0672_4bff_9738_3bc5682ea071.slice/crio-a7c2447f4aa3486628ad8f87b2e9265ab1dd86ead734b49ca836388dd5ce5411 WatchSource:0}: Error finding container a7c2447f4aa3486628ad8f87b2e9265ab1dd86ead734b49ca836388dd5ce5411: Status 404 returned error can't find the container with id a7c2447f4aa3486628ad8f87b2e9265ab1dd86ead734b49ca836388dd5ce5411
Sep 07 11:24:02.691361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.691337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:02.691460 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:02.691437    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:02.691512 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:02.691505    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls podName:fc1f1428-c3e4-4865-8bf8-e713111191a2 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:03.691486846 +0000 UTC m=+75.166334994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-v725n" (UID: "fc1f1428-c3e4-4865-8bf8-e713111191a2") : secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:02.743017 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:02.742977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx"]
Sep 07 11:24:02.746789 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:02.746759    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383ede9c_a994_4f54_88f9_c4647a3e6c8d.slice/crio-0c22060394f38d9ab4c8b67eb1cf2ba16ae0934817984075c54f59523fc804e8 WatchSource:0}: Error finding container 0c22060394f38d9ab4c8b67eb1cf2ba16ae0934817984075c54f59523fc804e8: Status 404 returned error can't find the container with id 0c22060394f38d9ab4c8b67eb1cf2ba16ae0934817984075c54f59523fc804e8
Sep 07 11:24:03.255772 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.255719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" event={"ID":"7954bc9a-b5a5-4829-aed7-d4004433d11c","Type":"ContainerStarted","Data":"545804e181c5e0b803753dd4f1f7f5bd97bfe7888880a6e587c939466dd55437"}
Sep 07 11:24:03.257647 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.257608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl" event={"ID":"61f0563e-6e58-46dc-be37-2e67461139c4","Type":"ContainerStarted","Data":"241a1dd38225914c17ac2cee824742fc4f1e264b32960f88c68d5c10504b6665"}
Sep 07 11:24:03.257788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.257670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl" event={"ID":"61f0563e-6e58-46dc-be37-2e67461139c4","Type":"ContainerStarted","Data":"de0fa0ae04f84a435f4c3b0d3124e75b0914be6059994946e4e4e36745c90744"}
Sep 07 11:24:03.259080 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.259044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx" event={"ID":"383ede9c-a994-4f54-88f9-c4647a3e6c8d","Type":"ContainerStarted","Data":"0c22060394f38d9ab4c8b67eb1cf2ba16ae0934817984075c54f59523fc804e8"}
Sep 07 11:24:03.260345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.260318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6" event={"ID":"4c3b1517-0672-4bff-9738-3bc5682ea071","Type":"ContainerStarted","Data":"a7c2447f4aa3486628ad8f87b2e9265ab1dd86ead734b49ca836388dd5ce5411"}
Sep 07 11:24:03.261723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.261700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt" event={"ID":"7a872110-863f-486c-b6d6-ff742fa69a5f","Type":"ContainerStarted","Data":"1ef3ad7b3ebf937fce26464c7cee38c5bc0a1ef519f79ae43563179ae4031a59"}
Sep 07 11:24:03.274627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.274575    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-vhznl" podStartSLOduration=2.274558063 podStartE2EDuration="2.274558063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:03.274302991 +0000 UTC m=+74.749151165" watchObservedRunningTime="2026-09-07 11:24:03.274558063 +0000 UTC m=+74.749406213"
Sep 07 11:24:03.699459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.699415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:03.699673 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:03.699591    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:03.699673 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:03.699667    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls podName:fc1f1428-c3e4-4865-8bf8-e713111191a2 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:05.699647285 +0000 UTC m=+77.174495431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-v725n" (UID: "fc1f1428-c3e4-4865-8bf8-e713111191a2") : secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:03.902968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:03.902928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:04.003234 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:04.003081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:05.716211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:05.716173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:05.716711 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:05.716337    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:05.716711 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:05.716427    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls podName:fc1f1428-c3e4-4865-8bf8-e713111191a2 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.716412116 +0000 UTC m=+81.191260243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-v725n" (UID: "fc1f1428-c3e4-4865-8bf8-e713111191a2") : secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:07.276772 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.276728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6" event={"ID":"4c3b1517-0672-4bff-9738-3bc5682ea071","Type":"ContainerStarted","Data":"7ebd45e41a9890f6febbfc015b5aa0d76186998651a36851668febce4fcae304"}
Sep 07 11:24:07.279098 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.279042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt" event={"ID":"7a872110-863f-486c-b6d6-ff742fa69a5f","Type":"ContainerStarted","Data":"294530f686c87ce84363c8726525fb93be6de60f0f3c7640636ecc1abced51bd"}
Sep 07 11:24:07.284013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.281378    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/0.log"
Sep 07 11:24:07.284013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.281415    2578 generic.go:356] "Generic (PLEG): container finished" podID="7954bc9a-b5a5-4829-aed7-d4004433d11c" containerID="ea6024be6d6643d6763761cef777147d87e6f9ddb74b146fa7d4bb269ad60a3e" exitCode=255
Sep 07 11:24:07.284013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.281661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" event={"ID":"7954bc9a-b5a5-4829-aed7-d4004433d11c","Type":"ContainerDied","Data":"ea6024be6d6643d6763761cef777147d87e6f9ddb74b146fa7d4bb269ad60a3e"}
Sep 07 11:24:07.284013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.281776    2578 scope.go:117] "RemoveContainer" containerID="ea6024be6d6643d6763761cef777147d87e6f9ddb74b146fa7d4bb269ad60a3e"
Sep 07 11:24:07.284013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.283793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx" event={"ID":"383ede9c-a994-4f54-88f9-c4647a3e6c8d","Type":"ContainerStarted","Data":"450f0ddc86c0c5d0516a97447d4cf1fcd72344db5b6ff7e78b5fdab6c7162828"}
Sep 07 11:24:07.295106 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.294565    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6" podStartSLOduration=1.606124069 podStartE2EDuration="5.294548246s" totalImagesPullingTime="3.688424177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:07.293827484 +0000 UTC m=+78.768675662" watchObservedRunningTime="2026-09-07 11:24:07.294548246 +0000 UTC m=+78.769396396"
Sep 07 11:24:07.341053 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.340990    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt" podStartSLOduration=2.552605142 podStartE2EDuration="6.340969372s" totalImagesPullingTime="3.78836423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:07.339282888 +0000 UTC m=+78.814131038" watchObservedRunningTime="2026-09-07 11:24:07.340969372 +0000 UTC m=+78.815817519"
Sep 07 11:24:07.361146 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.361101    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx" podStartSLOduration=1.732981045 podStartE2EDuration="5.361083184s" totalImagesPullingTime="3.628102139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:07.359878402 +0000 UTC m=+78.834726579" watchObservedRunningTime="2026-09-07 11:24:07.361083184 +0000 UTC m=+78.835931333"
Sep 07 11:24:07.375221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.375160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:07.397221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.396309    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2qc9p"]
Sep 07 11:24:07.399298 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.399277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.401653 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.401632    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6gln4\""
Sep 07 11:24:07.401944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.401889    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 11:24:07.402029 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.401976    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 11:24:07.411923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.411845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qc9p"]
Sep 07 11:24:07.430776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.430743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-data-volume\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.430864 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.430814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-crio-socket\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.430921 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.430885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.430975 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.430928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.431065 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.431040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscnp\" (UniqueName: \"kubernetes.io/projected/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-api-access-sscnp\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.447955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.447933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:07.531456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.531399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-data-volume\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.531456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.531441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-crio-socket\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.531603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.531528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-crio-socket\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.531603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.531559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.531603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.531584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.531719 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.531629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscnp\" (UniqueName: \"kubernetes.io/projected/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-api-access-sscnp\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.531719 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:07.531671    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:07.531802 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:07.531725    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:07.531802 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:07.531740    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:08.031721189 +0000 UTC m=+79.506569335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:07.531802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.531768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-data-volume\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.531802 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:07.531790    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:08.031771922 +0000 UTC m=+79.506620065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:07.539862 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.539829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscnp\" (UniqueName: \"kubernetes.io/projected/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-api-access-sscnp\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:07.747693 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.747656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:07.806671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.806608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:07.986182 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:07.986149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:08.036005 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.035969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:08.036198 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.036016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:08.036198 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:08.036085    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:08.036198 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:08.036157    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.036137752 +0000 UTC m=+80.510985898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:08.036198 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:08.036180    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:08.036360 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:08.036221    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.0362092 +0000 UTC m=+80.511057334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:08.072210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.072142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:08.131348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.131326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:08.228936 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.228907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:08.290970 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.290946    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:24:08.291361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.291348    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/0.log"
Sep 07 11:24:08.291411 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.291379    2578 generic.go:356] "Generic (PLEG): container finished" podID="7954bc9a-b5a5-4829-aed7-d4004433d11c" containerID="93d8a6d09367c05a23c13b2cedad6345d5c3cdad77628d1db0f0ab4b6e57f9cf" exitCode=255
Sep 07 11:24:08.291484 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.291451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" event={"ID":"7954bc9a-b5a5-4829-aed7-d4004433d11c","Type":"ContainerDied","Data":"93d8a6d09367c05a23c13b2cedad6345d5c3cdad77628d1db0f0ab4b6e57f9cf"}
Sep 07 11:24:08.291551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.291539    2578 scope.go:117] "RemoveContainer" containerID="ea6024be6d6643d6763761cef777147d87e6f9ddb74b146fa7d4bb269ad60a3e"
Sep 07 11:24:08.291729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.291704    2578 scope.go:117] "RemoveContainer" containerID="93d8a6d09367c05a23c13b2cedad6345d5c3cdad77628d1db0f0ab4b6e57f9cf"
Sep 07 11:24:08.291923 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:08.291896    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5xm2q_openshift-console-operator(7954bc9a-b5a5-4829-aed7-d4004433d11c)\"" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" podUID="7954bc9a-b5a5-4829-aed7-d4004433d11c"
Sep 07 11:24:08.457764 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.457732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:08.526004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:08.525976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:09.044561 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:09.044531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:09.044705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:09.044567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:09.044705 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:09.044653    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:09.044705 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:09.044674    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:09.044804 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:09.044714    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:11.044697833 +0000 UTC m=+82.519545978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:09.044804 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:09.044727    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:11.04472143 +0000 UTC m=+82.519569557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:09.296386 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:09.296323    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:24:09.296731 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:09.296622    2578 scope.go:117] "RemoveContainer" containerID="93d8a6d09367c05a23c13b2cedad6345d5c3cdad77628d1db0f0ab4b6e57f9cf"
Sep 07 11:24:09.296778 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:09.296755    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5xm2q_openshift-console-operator(7954bc9a-b5a5-4829-aed7-d4004433d11c)\"" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" podUID="7954bc9a-b5a5-4829-aed7-d4004433d11c"
Sep 07 11:24:09.332122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:09.332099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hkpwd_50fe92fb-f492-45be-894d-7e4428d1985b/dns-node-resolver/0.log"
Sep 07 11:24:09.748174 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:09.748142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:09.748319 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:09.748252    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:09.748319 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:09.748304    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls podName:fc1f1428-c3e4-4865-8bf8-e713111191a2 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:17.748292058 +0000 UTC m=+89.223140184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-v725n" (UID: "fc1f1428-c3e4-4865-8bf8-e713111191a2") : secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:10.048460 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.048385    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4tdzh"]
Sep 07 11:24:10.050167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.050146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.052530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.052505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 07 11:24:10.052623 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.052568    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 07 11:24:10.053587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.053570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 07 11:24:10.053682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.053619    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 07 11:24:10.053682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.053634    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tzq4c\""
Sep 07 11:24:10.060885 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.060866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4tdzh"]
Sep 07 11:24:10.084164 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.084140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:10.148603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.148580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:10.150573 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.150554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a103fb8c-4b10-4169-8f99-1bd74a472671-signing-cabundle\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.150639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.150592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6frcq\" (UniqueName: \"kubernetes.io/projected/a103fb8c-4b10-4169-8f99-1bd74a472671-kube-api-access-6frcq\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.150680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.150637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a103fb8c-4b10-4169-8f99-1bd74a472671-signing-key\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.251321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.251295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a103fb8c-4b10-4169-8f99-1bd74a472671-signing-key\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.251421 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.251352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a103fb8c-4b10-4169-8f99-1bd74a472671-signing-cabundle\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.251421 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.251382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frcq\" (UniqueName: \"kubernetes.io/projected/a103fb8c-4b10-4169-8f99-1bd74a472671-kube-api-access-6frcq\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.251987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.251964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a103fb8c-4b10-4169-8f99-1bd74a472671-signing-cabundle\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.253666 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.253645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a103fb8c-4b10-4169-8f99-1bd74a472671-signing-key\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.259293 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.259274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frcq\" (UniqueName: \"kubernetes.io/projected/a103fb8c-4b10-4169-8f99-1bd74a472671-kube-api-access-6frcq\") pod \"service-ca-75d99c484-4tdzh\" (UID: \"a103fb8c-4b10-4169-8f99-1bd74a472671\") " pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.359325 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.359271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4tdzh"
Sep 07 11:24:10.533672 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.533646    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnkxf_ff53ccaf-e144-41af-8248-489446634ebd/node-ca/0.log"
Sep 07 11:24:10.623906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.623879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4tdzh"]
Sep 07 11:24:10.626840 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:10.626815    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda103fb8c_4b10_4169_8f99_1bd74a472671.slice/crio-0fc0cccdabedcae77f07b978d53beacd61d2712dda807f5408a4be5b236a45ee WatchSource:0}: Error finding container 0fc0cccdabedcae77f07b978d53beacd61d2712dda807f5408a4be5b236a45ee: Status 404 returned error can't find the container with id 0fc0cccdabedcae77f07b978d53beacd61d2712dda807f5408a4be5b236a45ee
Sep 07 11:24:10.932948 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:10.932875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2tfrr_21838826-4304-4ee2-a828-5e201e7ff2d9/serve-healthcheck-canary/0.log"
Sep 07 11:24:11.026678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.026645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:11.059733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.059700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:11.059878 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.059745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:11.059878 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:11.059817    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:11.059878 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:11.059878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:15.059863582 +0000 UTC m=+86.534711708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:11.060017 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:11.059879    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:11.060017 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:11.059934    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:15.059917621 +0000 UTC m=+86.534765762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:11.304550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.304478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4tdzh" event={"ID":"a103fb8c-4b10-4169-8f99-1bd74a472671","Type":"ContainerStarted","Data":"3d575d1492d111c627ba2221146d1d5310fd593c64b325bea0880e3b44135efe"}
Sep 07 11:24:11.304550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.304530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4tdzh" event={"ID":"a103fb8c-4b10-4169-8f99-1bd74a472671","Type":"ContainerStarted","Data":"0fc0cccdabedcae77f07b978d53beacd61d2712dda807f5408a4be5b236a45ee"}
Sep 07 11:24:11.320888 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.320834    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-4tdzh" podStartSLOduration=1.320816249 podStartE2EDuration="1.320816249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:11.319219566 +0000 UTC m=+82.794067734" watchObservedRunningTime="2026-09-07 11:24:11.320816249 +0000 UTC m=+82.795664401"
Sep 07 11:24:11.602045 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.601949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:11.701867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.701733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:11.801832 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.801802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:11.902333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:11.902303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:12.002756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:12.002723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:12.110260 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:12.110227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:12.296146 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:12.296066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:12.390036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:12.390008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:12.390181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:12.390060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:12.390428 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:12.390415    2578 scope.go:117] "RemoveContainer" containerID="93d8a6d09367c05a23c13b2cedad6345d5c3cdad77628d1db0f0ab4b6e57f9cf"
Sep 07 11:24:12.390652 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:12.390631    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5xm2q_openshift-console-operator(7954bc9a-b5a5-4829-aed7-d4004433d11c)\"" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" podUID="7954bc9a-b5a5-4829-aed7-d4004433d11c"
Sep 07 11:24:13.101383 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:13.101340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:13.494548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:13.494518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:13.899325 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:13.899297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:14.082829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:14.082804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:14.422842 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:14.422808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:14.697015 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:14.696944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:15.090841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:15.090774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:15.090967 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:15.090883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:15.090967 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:15.090924    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:15.090967 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:15.090948    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:15.091067 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:15.090991    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:23.090975071 +0000 UTC m=+94.565823200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:15.091067 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:15.091007    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls podName:99eb80d4-4fa5-4248-ab8c-9e64574e4f7a nodeName:}" failed. No retries permitted until 2026-09-07 11:24:23.090999146 +0000 UTC m=+94.565847272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qc9p" (UID: "99eb80d4-4fa5-4248-ab8c-9e64574e4f7a") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:15.274301 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:15.274272    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:15.338570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:15.338542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:15.497156 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:15.497127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:15.901773 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:15.901742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:16.088231 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:16.088198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:16.155581 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:16.155520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:16.579268 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:16.579193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:17.290324 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:17.290293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:17.688828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:17.688792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:17.793235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:17.793198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:17.809332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:17.809302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:17.809449 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:17.809431    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:17.809537 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:17.809524    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls podName:fc1f1428-c3e4-4865-8bf8-e713111191a2 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:33.809505447 +0000 UTC m=+105.284353578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-v725n" (UID: "fc1f1428-c3e4-4865-8bf8-e713111191a2") : secret "cluster-monitoring-operator-tls" not found
Sep 07 11:24:17.901241 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:17.901205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:18.092723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:18.092655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:18.499509 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:18.499478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:19.296872 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:19.296839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:19.498546 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:19.498513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:19.899912 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:19.899880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:20.093596 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:20.093556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:20.586667 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:20.586632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:20.691284 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:20.691254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:21.093110 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:21.093078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:21.495649 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:21.495598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:21.893384 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:21.893352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:22.096363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:22.096336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:22.294903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:22.294825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:22.705532 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:22.705504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:23.147810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:23.147783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:23.148167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:23.147822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:23.148346 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:23.148326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:23.150104 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:23.150082    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99eb80d4-4fa5-4248-ab8c-9e64574e4f7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qc9p\" (UID: \"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a\") " pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:23.309578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:23.309531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qc9p"
Sep 07 11:24:23.578454 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:23.578384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qc9p"]
Sep 07 11:24:23.581693 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:23.581662    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99eb80d4_4fa5_4248_ab8c_9e64574e4f7a.slice/crio-52d72f0d7aa4540b6e4814f28aa849abb53f2405944ca2c397fd6a5d47300368 WatchSource:0}: Error finding container 52d72f0d7aa4540b6e4814f28aa849abb53f2405944ca2c397fd6a5d47300368: Status 404 returned error can't find the container with id 52d72f0d7aa4540b6e4814f28aa849abb53f2405944ca2c397fd6a5d47300368
Sep 07 11:24:24.347677 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:24.347638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qc9p" event={"ID":"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a","Type":"ContainerStarted","Data":"edefa7895daa5cc5d6383060f5aba2fc6f38fbf865909f25bfb0f0d14c8c10d7"}
Sep 07 11:24:24.348016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:24.347702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qc9p" event={"ID":"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a","Type":"ContainerStarted","Data":"e417dbe02baf899698b79ab0fb0270cb306bc42f0562bcad69f7db1d4558359f"}
Sep 07 11:24:24.348016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:24.347729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qc9p" event={"ID":"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a","Type":"ContainerStarted","Data":"52d72f0d7aa4540b6e4814f28aa849abb53f2405944ca2c397fd6a5d47300368"}
Sep 07 11:24:24.954195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:24.954167    2578 scope.go:117] "RemoveContainer" containerID="93d8a6d09367c05a23c13b2cedad6345d5c3cdad77628d1db0f0ab4b6e57f9cf"
Sep 07 11:24:25.353572 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.353496    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:24:25.353572 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.353557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" event={"ID":"7954bc9a-b5a5-4829-aed7-d4004433d11c","Type":"ContainerStarted","Data":"a0e8381817088363838b863874303fbfc747e6f7827ad78a79bc7b969d2c2a62"}
Sep 07 11:24:25.354033 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.353942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:25.370676 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.370628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-5xm2q" podStartSLOduration=19.663200541 podStartE2EDuration="23.370611352s" totalImagesPullingTime="3.707410811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:25.369803465 +0000 UTC m=+96.844651651" watchObservedRunningTime="2026-09-07 11:24:25.370611352 +0000 UTC m=+96.845459501"
Sep 07 11:24:25.666662 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.666627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-5xm2q"
Sep 07 11:24:25.767428 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.767402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:24:25.769315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.769294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52727a38-dbef-437a-96ec-4401f1a6292e-metrics-tls\") pod \"dns-default-tv695\" (UID: \"52727a38-dbef-437a-96ec-4401f1a6292e\") " pod="openshift-dns/dns-default-tv695"
Sep 07 11:24:25.775687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.775669    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ph89j\""
Sep 07 11:24:25.783830 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.783783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tv695"
Sep 07 11:24:25.845007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.844978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:25.944168 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:25.944093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.044034 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.044008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.075430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.075405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tv695"]
Sep 07 11:24:26.078261 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:26.078235    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52727a38_dbef_437a_96ec_4401f1a6292e.slice/crio-571f05ea71637afa06d5fbb392161180c58032ccbe866a77d5c9e575485dcbc5 WatchSource:0}: Error finding container 571f05ea71637afa06d5fbb392161180c58032ccbe866a77d5c9e575485dcbc5: Status 404 returned error can't find the container with id 571f05ea71637afa06d5fbb392161180c58032ccbe866a77d5c9e575485dcbc5
Sep 07 11:24:26.144063 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.144039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.324262 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.324198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.359018 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.358984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qc9p" event={"ID":"99eb80d4-4fa5-4248-ab8c-9e64574e4f7a","Type":"ContainerStarted","Data":"af5e0bb6f9662cb65cd93e044a5f2ac9e3b8bb11d2abdac90d193c465d13f0eb"}
Sep 07 11:24:26.361740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.361641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tv695" event={"ID":"52727a38-dbef-437a-96ec-4401f1a6292e","Type":"ContainerStarted","Data":"571f05ea71637afa06d5fbb392161180c58032ccbe866a77d5c9e575485dcbc5"}
Sep 07 11:24:26.379386 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.379342    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2qc9p" podStartSLOduration=17.371186704 podStartE2EDuration="19.379327557s" totalImagesPullingTime="2.008140853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:24:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:26.378556765 +0000 UTC m=+97.853404934" watchObservedRunningTime="2026-09-07 11:24:26.379327557 +0000 UTC m=+97.854175705"
Sep 07 11:24:26.533283 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.533245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.595490 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.595382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.726786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.726748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.827489 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.827449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:26.993682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:26.993642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:27.093845 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:27.093813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:27.333723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:27.333638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:27.547907 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:27.547870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:27.700108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:27.700066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:28.370936 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:28.370899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tv695" event={"ID":"52727a38-dbef-437a-96ec-4401f1a6292e","Type":"ContainerStarted","Data":"a848d7c9408a31378eb4449a731a31c6bc4abe86197c1bf6c35e5f2de8aeaa72"}
Sep 07 11:24:28.371073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:28.370962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tv695" event={"ID":"52727a38-dbef-437a-96ec-4401f1a6292e","Type":"ContainerStarted","Data":"6e058dcfe7c1fc0fa3b7da202aa340623bdf236340be2f9a99d853ea1c988138"}
Sep 07 11:24:28.371073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:28.371009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tv695"
Sep 07 11:24:28.388348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:28.388303    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tv695" podStartSLOduration=65.597886982 podStartE2EDuration="1m7.388290184s" totalImagesPullingTime="1.790403202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:28.388189558 +0000 UTC m=+99.863037727" watchObservedRunningTime="2026-09-07 11:24:28.388290184 +0000 UTC m=+99.863138370"
Sep 07 11:24:28.560759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:28.560729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:28.649270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:28.649202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:29.767761 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.766099    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"]
Sep 07 11:24:29.769446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.769416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:29.772563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.772537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 07 11:24:29.773792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.773768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2twbc\""
Sep 07 11:24:29.773908 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.773789    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 11:24:29.773908 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.773803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 11:24:29.773908 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.773819    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 11:24:29.781405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.781381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"]
Sep 07 11:24:29.847255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.847226    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-9xkx5"]
Sep 07 11:24:29.849143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.849127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-9xkx5"
Sep 07 11:24:29.851639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.851623    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4v8kx\""
Sep 07 11:24:29.851920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.851906    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 11:24:29.852008 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.851989    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 11:24:29.859181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.859162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:29.862588 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.862569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-9xkx5"]
Sep 07 11:24:29.895266 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.895246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7c5\" (UniqueName: \"kubernetes.io/projected/043202bc-f871-4192-9a8d-4bf4f0b8f662-kube-api-access-tc7c5\") pod \"downloads-85d98c998f-9xkx5\" (UID: \"043202bc-f871-4192-9a8d-4bf4f0b8f662\") " pod="openshift-console/downloads-85d98c998f-9xkx5"
Sep 07 11:24:29.895363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.895318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6cd46e75-b5d7-4512-92cf-f5cf4b20fb86-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm\" (UID: \"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:29.895363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.895346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gfxb\" (UniqueName: \"kubernetes.io/projected/6cd46e75-b5d7-4512-92cf-f5cf4b20fb86-kube-api-access-5gfxb\") pod \"managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm\" (UID: \"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:29.959108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.959066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:29.995935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.995904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7c5\" (UniqueName: \"kubernetes.io/projected/043202bc-f871-4192-9a8d-4bf4f0b8f662-kube-api-access-tc7c5\") pod \"downloads-85d98c998f-9xkx5\" (UID: \"043202bc-f871-4192-9a8d-4bf4f0b8f662\") " pod="openshift-console/downloads-85d98c998f-9xkx5"
Sep 07 11:24:29.996049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.996000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6cd46e75-b5d7-4512-92cf-f5cf4b20fb86-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm\" (UID: \"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:29.996049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.996034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gfxb\" (UniqueName: \"kubernetes.io/projected/6cd46e75-b5d7-4512-92cf-f5cf4b20fb86-kube-api-access-5gfxb\") pod \"managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm\" (UID: \"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:29.998425 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:29.998404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6cd46e75-b5d7-4512-92cf-f5cf4b20fb86-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm\" (UID: \"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:30.006662 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.006641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gfxb\" (UniqueName: \"kubernetes.io/projected/6cd46e75-b5d7-4512-92cf-f5cf4b20fb86-kube-api-access-5gfxb\") pod \"managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm\" (UID: \"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:30.006762 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.006743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7c5\" (UniqueName: \"kubernetes.io/projected/043202bc-f871-4192-9a8d-4bf4f0b8f662-kube-api-access-tc7c5\") pod \"downloads-85d98c998f-9xkx5\" (UID: \"043202bc-f871-4192-9a8d-4bf4f0b8f662\") " pod="openshift-console/downloads-85d98c998f-9xkx5"
Sep 07 11:24:30.053615 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.053565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:30.087591 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.087569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"
Sep 07 11:24:30.157782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.157755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-9xkx5"
Sep 07 11:24:30.352843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.352775    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm"]
Sep 07 11:24:30.355497 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:30.355457    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd46e75_b5d7_4512_92cf_f5cf4b20fb86.slice/crio-5aabb00358af0ad3a755a90f1ab2b11ebf1ae90f818c9bbd514fac72860da147 WatchSource:0}: Error finding container 5aabb00358af0ad3a755a90f1ab2b11ebf1ae90f818c9bbd514fac72860da147: Status 404 returned error can't find the container with id 5aabb00358af0ad3a755a90f1ab2b11ebf1ae90f818c9bbd514fac72860da147
Sep 07 11:24:30.379261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.379233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm" event={"ID":"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86","Type":"ContainerStarted","Data":"5aabb00358af0ad3a755a90f1ab2b11ebf1ae90f818c9bbd514fac72860da147"}
Sep 07 11:24:30.429756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:30.429733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-9xkx5"]
Sep 07 11:24:30.432772 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:30.432745    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043202bc_f871_4192_9a8d_4bf4f0b8f662.slice/crio-eb4a572ec2fcec2aa09557866862b932dfde1bb1d863f3a79223adfd79fb6090 WatchSource:0}: Error finding container eb4a572ec2fcec2aa09557866862b932dfde1bb1d863f3a79223adfd79fb6090: Status 404 returned error can't find the container with id eb4a572ec2fcec2aa09557866862b932dfde1bb1d863f3a79223adfd79fb6090
Sep 07 11:24:31.388342 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:31.388305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-9xkx5" event={"ID":"043202bc-f871-4192-9a8d-4bf4f0b8f662","Type":"ContainerStarted","Data":"eb4a572ec2fcec2aa09557866862b932dfde1bb1d863f3a79223adfd79fb6090"}
Sep 07 11:24:33.398445 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:33.398406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm" event={"ID":"6cd46e75-b5d7-4512-92cf-f5cf4b20fb86","Type":"ContainerStarted","Data":"ed97be68399e25c50043f17334920cef305d7842ba88fd005bd108d4f23f7b02"}
Sep 07 11:24:33.415522 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:33.415448    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f4dc5c7d-rdmcm" podStartSLOduration=1.893106236 podStartE2EDuration="4.415427275s" totalImagesPullingTime="2.522321039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:33.413719066 +0000 UTC m=+104.888567216" watchObservedRunningTime="2026-09-07 11:24:33.415427275 +0000 UTC m=+104.890275427"
Sep 07 11:24:33.828847 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:33.828762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:33.831312 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:33.831286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc1f1428-c3e4-4865-8bf8-e713111191a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-v725n\" (UID: \"fc1f1428-c3e4-4865-8bf8-e713111191a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:33.883326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:33.883286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"
Sep 07 11:24:34.161902 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:34.161873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n"]
Sep 07 11:24:34.165555 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:34.165525    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1f1428_c3e4_4865_8bf8_e713111191a2.slice/crio-6b94b3e94eaabecc9ac6265cbf1023babfc22da470b3ca75740d4aa813cb44ce WatchSource:0}: Error finding container 6b94b3e94eaabecc9ac6265cbf1023babfc22da470b3ca75740d4aa813cb44ce: Status 404 returned error can't find the container with id 6b94b3e94eaabecc9ac6265cbf1023babfc22da470b3ca75740d4aa813cb44ce
Sep 07 11:24:34.403966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:34.403919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n" event={"ID":"fc1f1428-c3e4-4865-8bf8-e713111191a2","Type":"ContainerStarted","Data":"6b94b3e94eaabecc9ac6265cbf1023babfc22da470b3ca75740d4aa813cb44ce"}
Sep 07 11:24:34.557010 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:34.556928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:34.623310 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:34.623271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:36.380244 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.380201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:36.416232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.416190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n" event={"ID":"fc1f1428-c3e4-4865-8bf8-e713111191a2","Type":"ContainerStarted","Data":"1fd84266a4e0a242cb5f4612f8362d8f0a661f757c5137223955ccf535f14140"}
Sep 07 11:24:36.433167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.433110    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-v725n" podStartSLOduration=32.563438816 podStartE2EDuration="34.433091717s" totalImagesPullingTime="1.869652901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:36.432122665 +0000 UTC m=+107.906970843" watchObservedRunningTime="2026-09-07 11:24:36.433091717 +0000 UTC m=+107.907939866"
Sep 07 11:24:36.436012 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.435981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:36.530950 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.530916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:36.537024 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.536994    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"]
Sep 07 11:24:36.540240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.540220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:36.542830 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.542806    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 07 11:24:36.542925 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.542850    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ztb2k\""
Sep 07 11:24:36.547220 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.547199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"]
Sep 07 11:24:36.650732 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.650649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/233d9c7f-f265-4329-8870-f931167526ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dkcws\" (UID: \"233d9c7f-f265-4329-8870-f931167526ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:36.751199 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.751165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/233d9c7f-f265-4329-8870-f931167526ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dkcws\" (UID: \"233d9c7f-f265-4329-8870-f931167526ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:36.751372 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:36.751316    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 07 11:24:36.751444 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:36.751401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233d9c7f-f265-4329-8870-f931167526ef-tls-certificates podName:233d9c7f-f265-4329-8870-f931167526ef nodeName:}" failed. No retries permitted until 2026-09-07 11:24:37.251380332 +0000 UTC m=+108.726228480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/233d9c7f-f265-4329-8870-f931167526ef-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-dkcws" (UID: "233d9c7f-f265-4329-8870-f931167526ef") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 07 11:24:36.818278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:36.818251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:37.256102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:37.256061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/233d9c7f-f265-4329-8870-f931167526ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dkcws\" (UID: \"233d9c7f-f265-4329-8870-f931167526ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:37.258955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:37.258924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/233d9c7f-f265-4329-8870-f931167526ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dkcws\" (UID: \"233d9c7f-f265-4329-8870-f931167526ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:37.454827 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:37.454792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:37.731611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:37.731586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"]
Sep 07 11:24:37.734066 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:37.734036    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233d9c7f_f265_4329_8870_f931167526ef.slice/crio-a5eb3534ac8d903c17d1d88a77e7c67e2e427d4d67bad141d1723c2cd8959c58 WatchSource:0}: Error finding container a5eb3534ac8d903c17d1d88a77e7c67e2e427d4d67bad141d1723c2cd8959c58: Status 404 returned error can't find the container with id a5eb3534ac8d903c17d1d88a77e7c67e2e427d4d67bad141d1723c2cd8959c58
Sep 07 11:24:38.377599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.377568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tv695"
Sep 07 11:24:38.425999 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.425962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws" event={"ID":"233d9c7f-f265-4329-8870-f931167526ef","Type":"ContainerStarted","Data":"a5eb3534ac8d903c17d1d88a77e7c67e2e427d4d67bad141d1723c2cd8959c58"}
Sep 07 11:24:38.960668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.960560    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-858967fc6-c8ts7"]
Sep 07 11:24:38.970862 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.970796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:38.980393 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.980365    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858967fc6-c8ts7"]
Sep 07 11:24:38.980547 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.980496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:38.983167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.983141    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 11:24:38.984137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.983948    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 11:24:38.984137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.983990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 11:24:38.984137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.983993    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 11:24:38.984137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.984122    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k84wv\""
Sep 07 11:24:38.984413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.984146    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 11:24:38.993829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:38.992035    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 11:24:39.033846 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.033800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:39.072816 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.072785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-trusted-ca-bundle\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.072976 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.072831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-console-config\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.072976 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.072890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-oauth-serving-cert\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.072976 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.072919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5d4\" (UniqueName: \"kubernetes.io/projected/b49784ee-d471-42f2-af45-50befc4d82c7-kube-api-access-df5d4\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.072976 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.072952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-service-ca\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.073191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.073057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-serving-cert\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.073191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.073107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-oauth-config\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.173574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.173525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-trusted-ca-bundle\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.173574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.173576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-console-config\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.173812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.173623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-oauth-serving-cert\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.173812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.173664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5d4\" (UniqueName: \"kubernetes.io/projected/b49784ee-d471-42f2-af45-50befc4d82c7-kube-api-access-df5d4\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.173812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.173710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-service-ca\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.173812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.173777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-serving-cert\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.173812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.173801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-oauth-config\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.174933 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.174892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-service-ca\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.175051 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.174893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-oauth-serving-cert\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.175051 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.175001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-trusted-ca-bundle\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.175498 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.175424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-console-config\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.176774 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.176751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-oauth-config\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.178043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.178018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-serving-cert\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.183779 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.183746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5d4\" (UniqueName: \"kubernetes.io/projected/b49784ee-d471-42f2-af45-50befc4d82c7-kube-api-access-df5d4\") pod \"console-858967fc6-c8ts7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") " pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.293898 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.293825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:39.584805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.584626    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858967fc6-c8ts7"]
Sep 07 11:24:39.647250 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:39.647211    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49784ee_d471_42f2_af45_50befc4d82c7.slice/crio-5387a358f2b377c979fdffd9cae228bb14858a8e7b78be3cc41b112f1db1fcc0 WatchSource:0}: Error finding container 5387a358f2b377c979fdffd9cae228bb14858a8e7b78be3cc41b112f1db1fcc0: Status 404 returned error can't find the container with id 5387a358f2b377c979fdffd9cae228bb14858a8e7b78be3cc41b112f1db1fcc0
Sep 07 11:24:39.961501 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:39.961443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:40.028356 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.028319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:40.436259 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.436210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws" event={"ID":"233d9c7f-f265-4329-8870-f931167526ef","Type":"ContainerStarted","Data":"52e6833d098afe80732ac317e10e277883755619788e00e4b98a56446176eeb3"}
Sep 07 11:24:40.436458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.436422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:40.437729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.437700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858967fc6-c8ts7" event={"ID":"b49784ee-d471-42f2-af45-50befc4d82c7","Type":"ContainerStarted","Data":"5387a358f2b377c979fdffd9cae228bb14858a8e7b78be3cc41b112f1db1fcc0"}
Sep 07 11:24:40.442432 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.442410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws"
Sep 07 11:24:40.452997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.452948    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dkcws" podStartSLOduration=2.516180478 podStartE2EDuration="4.452935285s" totalImagesPullingTime="1.936754807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:40.4518967 +0000 UTC m=+111.926744871" watchObservedRunningTime="2026-09-07 11:24:40.452935285 +0000 UTC m=+111.927783433"
Sep 07 11:24:40.563493 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.563413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:40.604497 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.604455    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"]
Sep 07 11:24:40.612853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.612832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.615173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.615150    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 07 11:24:40.615281 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.615176    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 07 11:24:40.615281 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.615227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 11:24:40.615494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.615435    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qbjc9\""
Sep 07 11:24:40.617747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.617727    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"]
Sep 07 11:24:40.631997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.631969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:40.687235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.687169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.687235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.687218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgdr\" (UniqueName: \"kubernetes.io/projected/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-kube-api-access-vjgdr\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.687424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.687248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.687424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.687276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.788655 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.788616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.788824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.788669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdr\" (UniqueName: \"kubernetes.io/projected/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-kube-api-access-vjgdr\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.788824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.788778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.788824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.788812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.789459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.789432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.791571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.791543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.791744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.791727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.796248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.796223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgdr\" (UniqueName: \"kubernetes.io/projected/1d5e3d1c-41be-47fc-9a87-d6af40e293e3-kube-api-access-vjgdr\") pod \"prometheus-operator-6f4946db56-sdsp8\" (UID: \"1d5e3d1c-41be-47fc-9a87-d6af40e293e3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:40.801795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.801769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:40.924681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:40.924651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"
Sep 07 11:24:42.467177 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:42.467133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:42.525988 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:42.525933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:43.513548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:43.513516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:43.614632 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:43.614589    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:43.763015 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:43.762977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:43.823109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:43.823023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:47.250623 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.250487    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-sdsp8"]
Sep 07 11:24:47.254332 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:47.254285    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5e3d1c_41be_47fc_9a87_d6af40e293e3.slice/crio-d05a10c1200e1c7e1b0948b39acc52665a9b169274c51910262eed134b7c1af4 WatchSource:0}: Error finding container d05a10c1200e1c7e1b0948b39acc52665a9b169274c51910262eed134b7c1af4: Status 404 returned error can't find the container with id d05a10c1200e1c7e1b0948b39acc52665a9b169274c51910262eed134b7c1af4
Sep 07 11:24:47.466594 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.466508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8" event={"ID":"1d5e3d1c-41be-47fc-9a87-d6af40e293e3","Type":"ContainerStarted","Data":"d05a10c1200e1c7e1b0948b39acc52665a9b169274c51910262eed134b7c1af4"}
Sep 07 11:24:47.469083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.469046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-9xkx5" event={"ID":"043202bc-f871-4192-9a8d-4bf4f0b8f662","Type":"ContainerStarted","Data":"bd8317d5ec757f6b7705f7761ebba0ad06fc551bbe463f7569359a568d5fcfb8"}
Sep 07 11:24:47.470088 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.470053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-9xkx5"
Sep 07 11:24:47.481729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.481695    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-9xkx5"
Sep 07 11:24:47.493229 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.492122    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-9xkx5" podStartSLOduration=1.8679165009999998 podStartE2EDuration="18.492105898s" totalImagesPullingTime="16.624189397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:47.490756054 +0000 UTC m=+118.965604227" watchObservedRunningTime="2026-09-07 11:24:47.492105898 +0000 UTC m=+118.966954047"
Sep 07 11:24:47.521535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.521419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:47.613073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:47.613027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:49.187759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:49.187718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:49.291213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:49.291121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:49.444761 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:49.444654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:49.574891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:49.574646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:49.683705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:49.683663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:50.170136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:50.170026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:50.273095 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:50.273060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:51.241371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:51.241284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:51.489454 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:51.489409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8" event={"ID":"1d5e3d1c-41be-47fc-9a87-d6af40e293e3","Type":"ContainerStarted","Data":"a22661af9ff6ac27ad61e21d2fd0cf78081642f5a6f999833f1e08b56c9be3a8"}
Sep 07 11:24:51.489931 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:51.489513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8" event={"ID":"1d5e3d1c-41be-47fc-9a87-d6af40e293e3","Type":"ContainerStarted","Data":"4ea99b4aa2da352adc2f87682c5c499d74ccbec026ae42bf2b0a2102c68070ca"}
Sep 07 11:24:51.491347 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:51.491315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858967fc6-c8ts7" event={"ID":"b49784ee-d471-42f2-af45-50befc4d82c7","Type":"ContainerStarted","Data":"5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17"}
Sep 07 11:24:51.507676 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:51.507599    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-sdsp8" podStartSLOduration=7.895729907 podStartE2EDuration="11.507583055s" totalImagesPullingTime="3.611853148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:51.506700573 +0000 UTC m=+122.981548748" watchObservedRunningTime="2026-09-07 11:24:51.507583055 +0000 UTC m=+122.982431208"
Sep 07 11:24:51.527417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:51.527361    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-858967fc6-c8ts7" podStartSLOduration=2.335166276 podStartE2EDuration="13.527343366s" totalImagesPullingTime="11.19217709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:51.526634745 +0000 UTC m=+123.001482919" watchObservedRunningTime="2026-09-07 11:24:51.527343366 +0000 UTC m=+123.002191514"
Sep 07 11:24:52.043400 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.043364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:52.108081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.108041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:52.591520 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.591488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:52.696889 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.696849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:52.792263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.792220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:52.943965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.943925    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"]
Sep 07 11:24:52.971339 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.971311    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vbvs9"]
Sep 07 11:24:52.973232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.973207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:52.977738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.977715    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6nqdk\""
Sep 07 11:24:52.978012 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.977942    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 07 11:24:52.978127 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.977942    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 07 11:24:52.988535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.987565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:52.991859 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.991325    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 11:24:52.991859 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.991567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 11:24:52.991859 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.991765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wb972\""
Sep 07 11:24:52.993195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.992686    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 11:24:52.996081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:52.996062    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"]
Sep 07 11:24:53.015515 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.015490    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"]
Sep 07 11:24:53.015639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.015544    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"]
Sep 07 11:24:53.015639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.015576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.019430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.019370    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 07 11:24:53.019430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.019387    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 07 11:24:53.019615 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.019599    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 07 11:24:53.019769 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.019714    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5xmg8\""
Sep 07 11:24:53.094309 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.094267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00a2ba78-7842-49fb-b9a3-bf35a59f563b-metrics-client-ca\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.094592 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.094565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.094728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.094714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-sys\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.094857 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.094841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-wtmp\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.094980 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.094963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.095154 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfmsl\" (UniqueName: \"kubernetes.io/projected/00a2ba78-7842-49fb-b9a3-bf35a59f563b-kube-api-access-mfmsl\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.095265 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.095392 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-tls\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.095392 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-root\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.095392 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.095392 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84rv\" (UniqueName: \"kubernetes.io/projected/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-kube-api-access-f84rv\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.095619 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-textfile\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.095619 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.095476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.108879 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.108815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.196601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-wtmp\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.196769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2024fd-cec9-4813-a943-d892c6382240-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.196925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfmsl\" (UniqueName: \"kubernetes.io/projected/00a2ba78-7842-49fb-b9a3-bf35a59f563b-kube-api-access-mfmsl\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-tls\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b2024fd-cec9-4813-a943-d892c6382240-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-root\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84rv\" (UniqueName: \"kubernetes.io/projected/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-kube-api-access-f84rv\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-textfile\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.197539 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.198458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00a2ba78-7842-49fb-b9a3-bf35a59f563b-metrics-client-ca\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.198458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.198458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-sys\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.198458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.197508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7b7l\" (UniqueName: \"kubernetes.io/projected/4b2024fd-cec9-4813-a943-d892c6382240-kube-api-access-r7b7l\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.198458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.198004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-sys\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.198733 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:53.198495    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 07 11:24:53.198733 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:53.198560    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-tls podName:00a2ba78-7842-49fb-b9a3-bf35a59f563b nodeName:}" failed. No retries permitted until 2026-09-07 11:24:53.698540886 +0000 UTC m=+125.173389020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-tls") pod "node-exporter-vbvs9" (UID: "00a2ba78-7842-49fb-b9a3-bf35a59f563b") : secret "node-exporter-tls" not found
Sep 07 11:24:53.198839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.198772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-root\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.199055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.199023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-textfile\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.199148 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:53.199108    2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 07 11:24:53.199202 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:24:53.199148    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-tls podName:2a4d9c60-09a2-450c-88c2-e5fe77f4a24d nodeName:}" failed. No retries permitted until 2026-09-07 11:24:53.699135499 +0000 UTC m=+125.173983632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-fnmvd" (UID: "2a4d9c60-09a2-450c-88c2-e5fe77f4a24d") : secret "openshift-state-metrics-tls" not found
Sep 07 11:24:53.200342 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.199783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00a2ba78-7842-49fb-b9a3-bf35a59f563b-metrics-client-ca\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.200342 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.200181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.201678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.200803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.201678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.196877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-wtmp\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.206483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.204650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.212981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.212876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.212981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.212909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfmsl\" (UniqueName: \"kubernetes.io/projected/00a2ba78-7842-49fb-b9a3-bf35a59f563b-kube-api-access-mfmsl\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.213863 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.213816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84rv\" (UniqueName: \"kubernetes.io/projected/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-kube-api-access-f84rv\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.214806 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.214704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:53.298011 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.297967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b7l\" (UniqueName: \"kubernetes.io/projected/4b2024fd-cec9-4813-a943-d892c6382240-kube-api-access-r7b7l\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.298227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.298042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2024fd-cec9-4813-a943-d892c6382240-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.298227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.298108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.298227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.298146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b2024fd-cec9-4813-a943-d892c6382240-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.298227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.298175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.298227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.298216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.298889 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.298861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b2024fd-cec9-4813-a943-d892c6382240-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.299161 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.299106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2024fd-cec9-4813-a943-d892c6382240-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.299575 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.299548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.301840 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.301810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.302310 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.302263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b2024fd-cec9-4813-a943-d892c6382240-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.313290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.313001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:53.313290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.313095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7b7l\" (UniqueName: \"kubernetes.io/projected/4b2024fd-cec9-4813-a943-d892c6382240-kube-api-access-r7b7l\") pod \"kube-state-metrics-74cb57df7-fvxtx\" (UID: \"4b2024fd-cec9-4813-a943-d892c6382240\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.328731 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.328515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"
Sep 07 11:24:53.476294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.476030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:53.653884 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.653850    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx"]
Sep 07 11:24:53.656748 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:53.656721    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2024fd_cec9_4813_a943_d892c6382240.slice/crio-0d09b4ba37bfc6220241de12c3ec56255e8c59ed6f3851e5f51e54fb00ebabde WatchSource:0}: Error finding container 0d09b4ba37bfc6220241de12c3ec56255e8c59ed6f3851e5f51e54fb00ebabde: Status 404 returned error can't find the container with id 0d09b4ba37bfc6220241de12c3ec56255e8c59ed6f3851e5f51e54fb00ebabde
Sep 07 11:24:53.692455 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.692419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:53.704792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.703942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.704792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.704084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-tls\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.707627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.707602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00a2ba78-7842-49fb-b9a3-bf35a59f563b-node-exporter-tls\") pod \"node-exporter-vbvs9\" (UID: \"00a2ba78-7842-49fb-b9a3-bf35a59f563b\") " pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.707756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.707671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a4d9c60-09a2-450c-88c2-e5fe77f4a24d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fnmvd\" (UID: \"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.887866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.887829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"
Sep 07 11:24:53.908315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.908246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbvs9"
Sep 07 11:24:53.920147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:53.920076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:53.949392 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:53.949314    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a2ba78_7842_49fb_b9a3_bf35a59f563b.slice/crio-82966c142834ef10127dcc5c0cc3904c0357f8432244433ff27f52f9926b00aa WatchSource:0}: Error finding container 82966c142834ef10127dcc5c0cc3904c0357f8432244433ff27f52f9926b00aa: Status 404 returned error can't find the container with id 82966c142834ef10127dcc5c0cc3904c0357f8432244433ff27f52f9926b00aa
Sep 07 11:24:54.019800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.019763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:54.085844 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.085765    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:54.207878 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.207807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd"]
Sep 07 11:24:54.211935 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:54.211906    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4d9c60_09a2_450c_88c2_e5fe77f4a24d.slice/crio-2c301099adcf9e60b77a6e6a0454918ff604890de7a3032d79c50c7e6f5070a7 WatchSource:0}: Error finding container 2c301099adcf9e60b77a6e6a0454918ff604890de7a3032d79c50c7e6f5070a7: Status 404 returned error can't find the container with id 2c301099adcf9e60b77a6e6a0454918ff604890de7a3032d79c50c7e6f5070a7
Sep 07 11:24:54.213357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.213331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:54.513141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.513097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd" event={"ID":"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d","Type":"ContainerStarted","Data":"7ace65d0e91e977c7f94648371dee2bd7810ce3cd78bc9b3b2008a41b790551b"}
Sep 07 11:24:54.513289 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.513165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd" event={"ID":"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d","Type":"ContainerStarted","Data":"9ee4f05ce3a12dc81f1121c0140f86638e84e20150d3bbb7d63e0e2d342ef7c3"}
Sep 07 11:24:54.513289 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.513200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd" event={"ID":"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d","Type":"ContainerStarted","Data":"2c301099adcf9e60b77a6e6a0454918ff604890de7a3032d79c50c7e6f5070a7"}
Sep 07 11:24:54.515092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.515063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx" event={"ID":"4b2024fd-cec9-4813-a943-d892c6382240","Type":"ContainerStarted","Data":"0d09b4ba37bfc6220241de12c3ec56255e8c59ed6f3851e5f51e54fb00ebabde"}
Sep 07 11:24:54.518455 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:54.518424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbvs9" event={"ID":"00a2ba78-7842-49fb-b9a3-bf35a59f563b","Type":"ContainerStarted","Data":"82966c142834ef10127dcc5c0cc3904c0357f8432244433ff27f52f9926b00aa"}
Sep 07 11:24:55.887828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:55.887789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:55.939167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:55.939135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:56.048309 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:56.048265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:56.150431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:56.150354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:56.994005 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:56.993976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:57.093236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.093154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:57.263863 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.263829    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"]
Sep 07 11:24:57.283680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.283650    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"]
Sep 07 11:24:57.283887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.283765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.286447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.286413    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 07 11:24:57.286705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.286679    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 07 11:24:57.287352 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.287326    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 07 11:24:57.287480 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.287392    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pd6xx\""
Sep 07 11:24:57.287480 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.287338    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 11:24:57.287480 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.287458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-49c0rb2gkjbep\""
Sep 07 11:24:57.339018 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.338984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-metrics-server-audit-profiles\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.339181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.339034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-secret-metrics-server-client-certs\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.339181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.339063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-secret-metrics-server-tls\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.339181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.339091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-audit-log\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.339181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.339108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-client-ca-bundle\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.339181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.339124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8bnp\" (UniqueName: \"kubernetes.io/projected/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-kube-api-access-z8bnp\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.339339 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.339212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.439693 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.439639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-metrics-server-audit-profiles\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.439906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.439713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-secret-metrics-server-client-certs\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.439906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.439732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-secret-metrics-server-tls\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.439906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.439755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-audit-log\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.439906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.439772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-client-ca-bundle\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.439906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.439793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8bnp\" (UniqueName: \"kubernetes.io/projected/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-kube-api-access-z8bnp\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.439906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.439846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.440223 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.440190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-audit-log\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.440668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.440642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.440756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.440666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-metrics-server-audit-profiles\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.442255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.442227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-secret-metrics-server-client-certs\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.442342 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.442328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-client-ca-bundle\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.442381 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.442358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-secret-metrics-server-tls\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.448550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.448457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8bnp\" (UniqueName: \"kubernetes.io/projected/bc3e7c79-39a0-4601-a6cd-6c6a5d424497-kube-api-access-z8bnp\") pod \"metrics-server-69df5b5b8d-ngz8w\" (UID: \"bc3e7c79-39a0-4601-a6cd-6c6a5d424497\") " pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.533601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.533566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd" event={"ID":"2a4d9c60-09a2-450c-88c2-e5fe77f4a24d","Type":"ContainerStarted","Data":"922ad6d2e6f5bf6b6c9122d041c144775c8a1606ea6a2423ef286c2e81e12905"}
Sep 07 11:24:57.535852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.535825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx" event={"ID":"4b2024fd-cec9-4813-a943-d892c6382240","Type":"ContainerStarted","Data":"63d4f575a5059cf21069e90b055294d9421a9d00415eebdcb9f879317a813346"}
Sep 07 11:24:57.535957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.535882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx" event={"ID":"4b2024fd-cec9-4813-a943-d892c6382240","Type":"ContainerStarted","Data":"b4dd6047d93be6c61952f1f473525924661d6d5747405bc5f7061f6ffa84fa98"}
Sep 07 11:24:57.535957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.535902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx" event={"ID":"4b2024fd-cec9-4813-a943-d892c6382240","Type":"ContainerStarted","Data":"a5c63b49800d84807fcdec1c17bef072c55d5b6dc0303af23f80b9fb3fec3a55"}
Sep 07 11:24:57.537356 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.537329    2578 generic.go:356] "Generic (PLEG): container finished" podID="00a2ba78-7842-49fb-b9a3-bf35a59f563b" containerID="cef496eabda1e5c841556591f7db3e356750939e6d848dadfcbdc048431767c3" exitCode=0
Sep 07 11:24:57.537453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.537368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbvs9" event={"ID":"00a2ba78-7842-49fb-b9a3-bf35a59f563b","Type":"ContainerDied","Data":"cef496eabda1e5c841556591f7db3e356750939e6d848dadfcbdc048431767c3"}
Sep 07 11:24:57.553395 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.553350    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fnmvd" podStartSLOduration=3.250221212 podStartE2EDuration="5.553331437s" totalImagesPullingTime="2.303110225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:57.552750181 +0000 UTC m=+129.027598349" watchObservedRunningTime="2026-09-07 11:24:57.553331437 +0000 UTC m=+129.028179585"
Sep 07 11:24:57.571328 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.571283    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fvxtx" podStartSLOduration=2.760765123 podStartE2EDuration="5.571269011s" totalImagesPullingTime="2.810503888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:57.570315568 +0000 UTC m=+129.045163736" watchObservedRunningTime="2026-09-07 11:24:57.571269011 +0000 UTC m=+129.046117159"
Sep 07 11:24:57.595119 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.595096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:24:57.746995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.746965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:57.759261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.759234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858967fc6-c8ts7"]
Sep 07 11:24:57.824335 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.824296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:57.908837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:57.908794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"]
Sep 07 11:24:57.911422 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:57.911392    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3e7c79_39a0_4601_a6cd_6c6a5d424497.slice/crio-5c36d16780f2ae036afb113749e573c0cbccd030abe19b5d6397fc26e69537be WatchSource:0}: Error finding container 5c36d16780f2ae036afb113749e573c0cbccd030abe19b5d6397fc26e69537be: Status 404 returned error can't find the container with id 5c36d16780f2ae036afb113749e573c0cbccd030abe19b5d6397fc26e69537be
Sep 07 11:24:58.149799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.149765    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79f946948-pvmxp"]
Sep 07 11:24:58.169053 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.169027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.172635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.172140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nbh4h\""
Sep 07 11:24:58.172635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.172155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 07 11:24:58.172635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.172238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 07 11:24:58.172635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.172142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 07 11:24:58.172635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.172442    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 07 11:24:58.172982 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.172865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 07 11:24:58.173655 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.173625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79f946948-pvmxp"]
Sep 07 11:24:58.176595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.176571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 07 11:24:58.245659 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-federate-client-tls\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.245659 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.245969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.245969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-telemeter-client-tls\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.245969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-secret-telemeter-client\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.245969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-serving-certs-ca-bundle\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.245969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-metrics-client-ca\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.245969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.245971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm7d2\" (UniqueName: \"kubernetes.io/projected/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-kube-api-access-gm7d2\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.312654 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.312618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:58.347024 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.346985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm7d2\" (UniqueName: \"kubernetes.io/projected/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-kube-api-access-gm7d2\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.347216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.347042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-federate-client-tls\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.347216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.347078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.347216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.347118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.347216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.347147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-telemeter-client-tls\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.347424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.347216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-secret-telemeter-client\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.347424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.347263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-serving-certs-ca-bundle\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.347424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.347305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-metrics-client-ca\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.348074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.348042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-serving-certs-ca-bundle\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.348193 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.348128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-metrics-client-ca\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.348193 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.348133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.349747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.349713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-federate-client-tls\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.349863 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.349754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.349863 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.349829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-secret-telemeter-client\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.349972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.349923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-telemeter-client-tls\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.357001 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.356979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm7d2\" (UniqueName: \"kubernetes.io/projected/c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5-kube-api-access-gm7d2\") pod \"telemeter-client-79f946948-pvmxp\" (UID: \"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5\") " pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.480957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.480868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79f946948-pvmxp"
Sep 07 11:24:58.542612 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.542575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w" event={"ID":"bc3e7c79-39a0-4601-a6cd-6c6a5d424497","Type":"ContainerStarted","Data":"5c36d16780f2ae036afb113749e573c0cbccd030abe19b5d6397fc26e69537be"}
Sep 07 11:24:58.545375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.545231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbvs9" event={"ID":"00a2ba78-7842-49fb-b9a3-bf35a59f563b","Type":"ContainerStarted","Data":"597664f622f4c8a9f3df2dc803113287797c152b3dd8bbdcf23e8ae424733495"}
Sep 07 11:24:58.545375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.545362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbvs9" event={"ID":"00a2ba78-7842-49fb-b9a3-bf35a59f563b","Type":"ContainerStarted","Data":"12e00cbd1f2f02dabfb0161c73608990a72c654769cf70aaa5f26ff2d8e2a746"}
Sep 07 11:24:58.567064 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.567012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vbvs9" podStartSLOduration=3.954197139 podStartE2EDuration="6.566991861s" totalImagesPullingTime="2.520181254s" totalInitContainerRuntime="92.613468ms" isStatefulPod=true podCreationTimestamp="2026-09-07 11:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:58.56632288 +0000 UTC m=+130.041171048" watchObservedRunningTime="2026-09-07 11:24:58.566991861 +0000 UTC m=+130.041840010"
Sep 07 11:24:58.649769 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.649715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:24:58.652394 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.652365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90bddad-de76-4653-8a1b-e3546fb5aa7f-metrics-certs\") pod \"network-metrics-daemon-lthxp\" (UID: \"a90bddad-de76-4653-8a1b-e3546fb5aa7f\") " pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:24:58.784886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.784766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79f946948-pvmxp"]
Sep 07 11:24:58.787924 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:58.787883    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f2e815_c78e_4bc6_a8c8_63f3f7a7def5.slice/crio-8eae7e75f8929a8c755d90170fca055fb4db90d2a8afc9172fecede01d525262 WatchSource:0}: Error finding container 8eae7e75f8929a8c755d90170fca055fb4db90d2a8afc9172fecede01d525262: Status 404 returned error can't find the container with id 8eae7e75f8929a8c755d90170fca055fb4db90d2a8afc9172fecede01d525262
Sep 07 11:24:58.880999 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.880337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qlhnv\""
Sep 07 11:24:58.888316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:58.888284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lthxp"
Sep 07 11:24:59.186922 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:59.186884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lthxp"]
Sep 07 11:24:59.195492 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:59.194813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:24:59.294261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:59.294226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:24:59.556233 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:24:59.556108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f946948-pvmxp" event={"ID":"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5","Type":"ContainerStarted","Data":"8eae7e75f8929a8c755d90170fca055fb4db90d2a8afc9172fecede01d525262"}
Sep 07 11:24:59.681723 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:24:59.681676    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90bddad_de76_4653_8a1b_e3546fb5aa7f.slice/crio-389bcebb831f3cdfc44dac61e62445fd66cd51ad11ad4d97f1342809e9d4dca9 WatchSource:0}: Error finding container 389bcebb831f3cdfc44dac61e62445fd66cd51ad11ad4d97f1342809e9d4dca9: Status 404 returned error can't find the container with id 389bcebb831f3cdfc44dac61e62445fd66cd51ad11ad4d97f1342809e9d4dca9
Sep 07 11:25:00.130922 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:00.130895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:00.232261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:00.232222    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:00.563146 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:00.563105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lthxp" event={"ID":"a90bddad-de76-4653-8a1b-e3546fb5aa7f","Type":"ContainerStarted","Data":"389bcebb831f3cdfc44dac61e62445fd66cd51ad11ad4d97f1342809e9d4dca9"}
Sep 07 11:25:00.565003 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:00.564972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w" event={"ID":"bc3e7c79-39a0-4601-a6cd-6c6a5d424497","Type":"ContainerStarted","Data":"06894236628fe95517242eca2636087f916f737a3ef702b078de2c00f5aab864"}
Sep 07 11:25:00.583185 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:00.583135    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w" podStartSLOduration=1.496353348 podStartE2EDuration="3.583120647s" totalImagesPullingTime="2.086767299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:25:00.581669619 +0000 UTC m=+132.056517787" watchObservedRunningTime="2026-09-07 11:25:00.583120647 +0000 UTC m=+132.057968796"
Sep 07 11:25:02.573834 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:02.573731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f946948-pvmxp" event={"ID":"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5","Type":"ContainerStarted","Data":"f64752dfddac88422dc5ff92d6797d0f45b96b382b641d127cc9faf5e53a2b66"}
Sep 07 11:25:02.575638 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:02.575608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lthxp" event={"ID":"a90bddad-de76-4653-8a1b-e3546fb5aa7f","Type":"ContainerStarted","Data":"4135b30ba1b5f7232231ad8ea10140a8deefd33f2c300bdc83757608c952e113"}
Sep 07 11:25:02.575789 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:02.575654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lthxp" event={"ID":"a90bddad-de76-4653-8a1b-e3546fb5aa7f","Type":"ContainerStarted","Data":"5df41584c2212d97a79c77771b974ef83d61085edea2612ce352d05217436a2a"}
Sep 07 11:25:02.593235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:02.593181    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lthxp" podStartSLOduration=131.381664446 podStartE2EDuration="2m13.593164824s" totalImagesPullingTime="2.211500378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:25:02.591521783 +0000 UTC m=+134.066369953" watchObservedRunningTime="2026-09-07 11:25:02.593164824 +0000 UTC m=+134.068012986"
Sep 07 11:25:02.684341 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:02.684309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:02.749387 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:02.749356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:04.587170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:04.587136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f946948-pvmxp" event={"ID":"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5","Type":"ContainerStarted","Data":"df0a3fc6454b5c2abf264b50f71d74d8c6686ea778728dce20048ba8147c3e91"}
Sep 07 11:25:04.587639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:04.587191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f946948-pvmxp" event={"ID":"c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5","Type":"ContainerStarted","Data":"3822f6f8ce5a1fa36da89a30448b4eaf0a5cdfb508a579b92a227c585cf2205e"}
Sep 07 11:25:04.610418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:04.610359    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79f946948-pvmxp" podStartSLOduration=1.777400299 podStartE2EDuration="6.610342419s" totalImagesPullingTime="4.83294212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:25:04.609143576 +0000 UTC m=+136.083991752" watchObservedRunningTime="2026-09-07 11:25:04.610342419 +0000 UTC m=+136.085190568"
Sep 07 11:25:11.993485 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:11.993428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:12.044474 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:12.044436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:15.246044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:15.246007    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:15.351952 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:15.351914    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:15.404231 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:15.404200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:17.595245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:17.595207    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:25:17.595661 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:17.595293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:25:22.659090 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:22.659053    2578 generic.go:356] "Generic (PLEG): container finished" podID="7a872110-863f-486c-b6d6-ff742fa69a5f" containerID="294530f686c87ce84363c8726525fb93be6de60f0f3c7640636ecc1abced51bd" exitCode=0
Sep 07 11:25:22.659550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:22.659127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt" event={"ID":"7a872110-863f-486c-b6d6-ff742fa69a5f","Type":"ContainerDied","Data":"294530f686c87ce84363c8726525fb93be6de60f0f3c7640636ecc1abced51bd"}
Sep 07 11:25:22.659550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:22.659492    2578 scope.go:117] "RemoveContainer" containerID="294530f686c87ce84363c8726525fb93be6de60f0f3c7640636ecc1abced51bd"
Sep 07 11:25:22.781169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:22.781112    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-858967fc6-c8ts7" podUID="b49784ee-d471-42f2-af45-50befc4d82c7" containerName="console" containerID="cri-o://5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17" gracePeriod=15
Sep 07 11:25:23.056324 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.056291    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:25:23.161380 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-oauth-serving-cert\") pod \"b49784ee-d471-42f2-af45-50befc4d82c7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") "
Sep 07 11:25:23.161380 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-console-config\") pod \"b49784ee-d471-42f2-af45-50befc4d82c7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") "
Sep 07 11:25:23.161380 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-service-ca\") pod \"b49784ee-d471-42f2-af45-50befc4d82c7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") "
Sep 07 11:25:23.161621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-trusted-ca-bundle\") pod \"b49784ee-d471-42f2-af45-50befc4d82c7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") "
Sep 07 11:25:23.161621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161432    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-oauth-config\") pod \"b49784ee-d471-42f2-af45-50befc4d82c7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") "
Sep 07 11:25:23.161621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-serving-cert\") pod \"b49784ee-d471-42f2-af45-50befc4d82c7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") "
Sep 07 11:25:23.161621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161549    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df5d4\" (UniqueName: \"kubernetes.io/projected/b49784ee-d471-42f2-af45-50befc4d82c7-kube-api-access-df5d4\") pod \"b49784ee-d471-42f2-af45-50befc4d82c7\" (UID: \"b49784ee-d471-42f2-af45-50befc4d82c7\") "
Sep 07 11:25:23.161919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-console-config" (OuterVolumeSpecName: "console-config") pod "b49784ee-d471-42f2-af45-50befc4d82c7" (UID: "b49784ee-d471-42f2-af45-50befc4d82c7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:25:23.162025 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.161978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-service-ca" (OuterVolumeSpecName: "service-ca") pod "b49784ee-d471-42f2-af45-50befc4d82c7" (UID: "b49784ee-d471-42f2-af45-50befc4d82c7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:25:23.162149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.162048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b49784ee-d471-42f2-af45-50befc4d82c7" (UID: "b49784ee-d471-42f2-af45-50befc4d82c7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:25:23.162149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.162100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b49784ee-d471-42f2-af45-50befc4d82c7" (UID: "b49784ee-d471-42f2-af45-50befc4d82c7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:25:23.163962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.163932    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b49784ee-d471-42f2-af45-50befc4d82c7" (UID: "b49784ee-d471-42f2-af45-50befc4d82c7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:25:23.164069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.163984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49784ee-d471-42f2-af45-50befc4d82c7-kube-api-access-df5d4" (OuterVolumeSpecName: "kube-api-access-df5d4") pod "b49784ee-d471-42f2-af45-50befc4d82c7" (UID: "b49784ee-d471-42f2-af45-50befc4d82c7"). InnerVolumeSpecName "kube-api-access-df5d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:25:23.164069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.164021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b49784ee-d471-42f2-af45-50befc4d82c7" (UID: "b49784ee-d471-42f2-af45-50befc4d82c7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:25:23.262648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.262614    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-oauth-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:25:23.262648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.262646    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-console-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:25:23.262648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.262655    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:25:23.262648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.262663    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b49784ee-d471-42f2-af45-50befc4d82c7-trusted-ca-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:25:23.262909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.262672    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-oauth-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:25:23.262909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.262680    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49784ee-d471-42f2-af45-50befc4d82c7-console-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:25:23.262909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.262688    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df5d4\" (UniqueName: \"kubernetes.io/projected/b49784ee-d471-42f2-af45-50befc4d82c7-kube-api-access-df5d4\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:25:23.664581 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.664549    2578 generic.go:356] "Generic (PLEG): container finished" podID="b49784ee-d471-42f2-af45-50befc4d82c7" containerID="5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17" exitCode=0
Sep 07 11:25:23.665006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.664598    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858967fc6-c8ts7"
Sep 07 11:25:23.665006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.664641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858967fc6-c8ts7" event={"ID":"b49784ee-d471-42f2-af45-50befc4d82c7","Type":"ContainerDied","Data":"5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17"}
Sep 07 11:25:23.665006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.664695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858967fc6-c8ts7" event={"ID":"b49784ee-d471-42f2-af45-50befc4d82c7","Type":"ContainerDied","Data":"5387a358f2b377c979fdffd9cae228bb14858a8e7b78be3cc41b112f1db1fcc0"}
Sep 07 11:25:23.665006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.664717    2578 scope.go:117] "RemoveContainer" containerID="5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17"
Sep 07 11:25:23.666788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.666763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cqztt" event={"ID":"7a872110-863f-486c-b6d6-ff742fa69a5f","Type":"ContainerStarted","Data":"3d12f42c2d0e488ac0ff4d2386de015fb6fa8b8bda44a086b6022a5132f2bdf8"}
Sep 07 11:25:23.674527 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.674511    2578 scope.go:117] "RemoveContainer" containerID="5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17"
Sep 07 11:25:23.674817 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:25:23.674788    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17\": container with ID starting with 5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17 not found: ID does not exist" containerID="5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17"
Sep 07 11:25:23.674891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.674829    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17"} err="failed to get container status \"5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17\": rpc error: code = NotFound desc = could not find container \"5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17\": container with ID starting with 5a08017bc1b7ff354589de9bad845fa0c0accdddefda9d9f3bad5ef38180cb17 not found: ID does not exist"
Sep 07 11:25:23.701854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.701826    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858967fc6-c8ts7"]
Sep 07 11:25:23.707421 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:23.707397    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858967fc6-c8ts7"]
Sep 07 11:25:24.959384 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:24.959352    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b49784ee-d471-42f2-af45-50befc4d82c7" path="/var/lib/kubelet/pods/b49784ee-d471-42f2-af45-50befc4d82c7/volumes"
Sep 07 11:25:32.703084 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:32.703049    2578 generic.go:356] "Generic (PLEG): container finished" podID="383ede9c-a994-4f54-88f9-c4647a3e6c8d" containerID="450f0ddc86c0c5d0516a97447d4cf1fcd72344db5b6ff7e78b5fdab6c7162828" exitCode=0
Sep 07 11:25:32.703564 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:32.703124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx" event={"ID":"383ede9c-a994-4f54-88f9-c4647a3e6c8d","Type":"ContainerDied","Data":"450f0ddc86c0c5d0516a97447d4cf1fcd72344db5b6ff7e78b5fdab6c7162828"}
Sep 07 11:25:32.703564 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:32.703500    2578 scope.go:117] "RemoveContainer" containerID="450f0ddc86c0c5d0516a97447d4cf1fcd72344db5b6ff7e78b5fdab6c7162828"
Sep 07 11:25:32.704678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:32.704656    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c3b1517-0672-4bff-9738-3bc5682ea071" containerID="7ebd45e41a9890f6febbfc015b5aa0d76186998651a36851668febce4fcae304" exitCode=0
Sep 07 11:25:32.704788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:32.704701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6" event={"ID":"4c3b1517-0672-4bff-9738-3bc5682ea071","Type":"ContainerDied","Data":"7ebd45e41a9890f6febbfc015b5aa0d76186998651a36851668febce4fcae304"}
Sep 07 11:25:32.705014 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:32.704999    2578 scope.go:117] "RemoveContainer" containerID="7ebd45e41a9890f6febbfc015b5aa0d76186998651a36851668febce4fcae304"
Sep 07 11:25:33.710110 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:33.710073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fpjjx" event={"ID":"383ede9c-a994-4f54-88f9-c4647a3e6c8d","Type":"ContainerStarted","Data":"e8875a3020b6cdae30459843be7f71f74b6d6fafaae36605e40dc53120d8de50"}
Sep 07 11:25:33.712043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:33.712020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z5hz6" event={"ID":"4c3b1517-0672-4bff-9738-3bc5682ea071","Type":"ContainerStarted","Data":"65651c7eda9e34529cae2afe174d13d53043dfdacbbfd7388e47377bf14d9dc3"}
Sep 07 11:25:35.681220 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:35.681183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:35.739352 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:35.739321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:25:37.601247 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:37.601212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:25:37.605207 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:25:37.605183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69df5b5b8d-ngz8w"
Sep 07 11:26:09.055706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:09.055661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:09.117903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:09.117875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:16.022778 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.022743    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d8f5d6cd9-wn77d"]
Sep 07 11:26:16.023242 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.023042    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b49784ee-d471-42f2-af45-50befc4d82c7" containerName="console"
Sep 07 11:26:16.023242 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.023054    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b49784ee-d471-42f2-af45-50befc4d82c7" containerName="console"
Sep 07 11:26:16.023242 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.023113    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b49784ee-d471-42f2-af45-50befc4d82c7" containerName="console"
Sep 07 11:26:16.025128 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.025105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.027257 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.027234    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k84wv\""
Sep 07 11:26:16.028311 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.028274    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 11:26:16.028311 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.028299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 11:26:16.028496 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.028355    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 11:26:16.028496 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.028356    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 11:26:16.028496 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.028436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 11:26:16.032661 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.032118    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 11:26:16.036044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.036021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8f5d6cd9-wn77d"]
Sep 07 11:26:16.070272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.070247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:16.080992 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.080972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-oauth-serving-cert\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.081102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.081000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-serving-cert\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.081102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.081036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-service-ca\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.081102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.081062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-oauth-config\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.081102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.081080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnh69\" (UniqueName: \"kubernetes.io/projected/4d92c80e-0c04-4553-9e01-08f40da9edf3-kube-api-access-hnh69\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.081281 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.081130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-trusted-ca-bundle\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.081281 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.081166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-config\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.121611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.121591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:16.182412 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.182386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnh69\" (UniqueName: \"kubernetes.io/projected/4d92c80e-0c04-4553-9e01-08f40da9edf3-kube-api-access-hnh69\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.182540 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.182414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-trusted-ca-bundle\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.182540 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.182433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-config\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.182733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.182575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-oauth-serving-cert\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.182733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.182607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-serving-cert\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.182733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.182646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-service-ca\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.182733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.182696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-oauth-config\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.183397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.183376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-trusted-ca-bundle\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.183503 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.183400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-oauth-serving-cert\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.183503 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.183437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-config\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.183503 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.183453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-service-ca\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.185194 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.185171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-oauth-config\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.185285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.185178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-serving-cert\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.190146 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.190124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnh69\" (UniqueName: \"kubernetes.io/projected/4d92c80e-0c04-4553-9e01-08f40da9edf3-kube-api-access-hnh69\") pod \"console-6d8f5d6cd9-wn77d\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") " pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.336060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.336007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:16.606038 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.605964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8f5d6cd9-wn77d"]
Sep 07 11:26:16.609530 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:26:16.609503    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d92c80e_0c04_4553_9e01_08f40da9edf3.slice/crio-be41d6783bf642d31bc4575ed48747ac5608f68425897c63b6d315db82ce6cb4 WatchSource:0}: Error finding container be41d6783bf642d31bc4575ed48747ac5608f68425897c63b6d315db82ce6cb4: Status 404 returned error can't find the container with id be41d6783bf642d31bc4575ed48747ac5608f68425897c63b6d315db82ce6cb4
Sep 07 11:26:16.888170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.888132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8f5d6cd9-wn77d" event={"ID":"4d92c80e-0c04-4553-9e01-08f40da9edf3","Type":"ContainerStarted","Data":"4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73"}
Sep 07 11:26:16.888334 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.888195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8f5d6cd9-wn77d" event={"ID":"4d92c80e-0c04-4553-9e01-08f40da9edf3","Type":"ContainerStarted","Data":"be41d6783bf642d31bc4575ed48747ac5608f68425897c63b6d315db82ce6cb4"}
Sep 07 11:26:16.905999 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:16.905945    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d8f5d6cd9-wn77d" podStartSLOduration=0.90593011 podStartE2EDuration="905.93011ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:26:16.904812694 +0000 UTC m=+208.379660861" watchObservedRunningTime="2026-09-07 11:26:16.90593011 +0000 UTC m=+208.380778259"
Sep 07 11:26:21.002138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:21.002093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:21.068657 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:21.068617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:26.336832 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:26.336714    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:26.336832 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:26.336785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:26.341563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:26.341531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:26.930154 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:26.930122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:26:27.013076 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:27.013043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:27.113716 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:27.113684    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:27.214060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:27.213981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:26:27.276631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:26:27.276601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:03.275627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:03.275594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:03.358782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:03.358748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:03.637713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:03.637663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:03.728697 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:03.728662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:38.542797 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:38.542762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:38.606813 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:38.606776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:39.339333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.339299    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"]
Sep 07 11:27:39.341731 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.341711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.344043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.344023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm44s\""
Sep 07 11:27:39.344043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.344030    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 11:27:39.344962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.344943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 11:27:39.350685 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.350667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"]
Sep 07 11:27:39.436276 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.436250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.436415 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.436283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8lg\" (UniqueName: \"kubernetes.io/projected/4320e956-8c02-4206-9eaf-c8c900b3c677-kube-api-access-dc8lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.436415 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.436354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.537375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.537347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.537517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.537402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.537517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.537434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8lg\" (UniqueName: \"kubernetes.io/projected/4320e956-8c02-4206-9eaf-c8c900b3c677-kube-api-access-dc8lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.537728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.537706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.537795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.537779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.546071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.546052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8lg\" (UniqueName: \"kubernetes.io/projected/4320e956-8c02-4206-9eaf-c8c900b3c677-kube-api-access-dc8lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.651170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.651145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:39.922704 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:39.922675    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"]
Sep 07 11:27:39.925138 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:27:39.925113    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4320e956_8c02_4206_9eaf_c8c900b3c677.slice/crio-00ab22a7e4547d0e87d25b3f203110f7188f51b4ca37fce31ab9159e75204b98 WatchSource:0}: Error finding container 00ab22a7e4547d0e87d25b3f203110f7188f51b4ca37fce31ab9159e75204b98: Status 404 returned error can't find the container with id 00ab22a7e4547d0e87d25b3f203110f7188f51b4ca37fce31ab9159e75204b98
Sep 07 11:27:40.207028 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:40.206947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8" event={"ID":"4320e956-8c02-4206-9eaf-c8c900b3c677","Type":"ContainerStarted","Data":"00ab22a7e4547d0e87d25b3f203110f7188f51b4ca37fce31ab9159e75204b98"}
Sep 07 11:27:42.276281 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:42.276246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:27:47.240409 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:47.240374    2578 generic.go:356] "Generic (PLEG): container finished" podID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerID="ebf6e46b7fc821c5737e3c9bd7f16387206b9642fce72dd9f0e9ff8329f785ab" exitCode=0
Sep 07 11:27:47.240881 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:47.240419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8" event={"ID":"4320e956-8c02-4206-9eaf-c8c900b3c677","Type":"ContainerDied","Data":"ebf6e46b7fc821c5737e3c9bd7f16387206b9642fce72dd9f0e9ff8329f785ab"}
Sep 07 11:27:49.265272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:49.265237    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:27:49.266302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:49.265984    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:27:49.270100 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:49.270081    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:27:49.270729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:49.270702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:27:49.277151 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:49.277135    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 11:27:50.254224 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:50.254191    2578 generic.go:356] "Generic (PLEG): container finished" podID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerID="dadec873beaf192b64ec02ee877ac729e7dcbbef3c9edd881e4cacac9439dbf9" exitCode=0
Sep 07 11:27:50.254386 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:50.254264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8" event={"ID":"4320e956-8c02-4206-9eaf-c8c900b3c677","Type":"ContainerDied","Data":"dadec873beaf192b64ec02ee877ac729e7dcbbef3c9edd881e4cacac9439dbf9"}
Sep 07 11:27:50.255234 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:50.255218    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:27:57.289683 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:57.289644    2578 generic.go:356] "Generic (PLEG): container finished" podID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerID="8ed8b1d9ed60e14a499cf145009d0a7f0517cbb3c6471216f40e348d28f4c6d1" exitCode=0
Sep 07 11:27:57.290040 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:57.289720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8" event={"ID":"4320e956-8c02-4206-9eaf-c8c900b3c677","Type":"ContainerDied","Data":"8ed8b1d9ed60e14a499cf145009d0a7f0517cbb3c6471216f40e348d28f4c6d1"}
Sep 07 11:27:58.411784 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.411762    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:58.499526 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.499496    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-util\") pod \"4320e956-8c02-4206-9eaf-c8c900b3c677\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") "
Sep 07 11:27:58.499673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.499534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-bundle\") pod \"4320e956-8c02-4206-9eaf-c8c900b3c677\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") "
Sep 07 11:27:58.499673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.499596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc8lg\" (UniqueName: \"kubernetes.io/projected/4320e956-8c02-4206-9eaf-c8c900b3c677-kube-api-access-dc8lg\") pod \"4320e956-8c02-4206-9eaf-c8c900b3c677\" (UID: \"4320e956-8c02-4206-9eaf-c8c900b3c677\") "
Sep 07 11:27:58.500177 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.500148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-bundle" (OuterVolumeSpecName: "bundle") pod "4320e956-8c02-4206-9eaf-c8c900b3c677" (UID: "4320e956-8c02-4206-9eaf-c8c900b3c677"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:27:58.501742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.501708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4320e956-8c02-4206-9eaf-c8c900b3c677-kube-api-access-dc8lg" (OuterVolumeSpecName: "kube-api-access-dc8lg") pod "4320e956-8c02-4206-9eaf-c8c900b3c677" (UID: "4320e956-8c02-4206-9eaf-c8c900b3c677"). InnerVolumeSpecName "kube-api-access-dc8lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:27:58.505380 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.505354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-util" (OuterVolumeSpecName: "util") pod "4320e956-8c02-4206-9eaf-c8c900b3c677" (UID: "4320e956-8c02-4206-9eaf-c8c900b3c677"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:27:58.600162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.600105    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:27:58.600162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.600127    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4320e956-8c02-4206-9eaf-c8c900b3c677-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:27:58.600162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:58.600137    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8lg\" (UniqueName: \"kubernetes.io/projected/4320e956-8c02-4206-9eaf-c8c900b3c677-kube-api-access-dc8lg\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:27:59.299662 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:59.299584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8" event={"ID":"4320e956-8c02-4206-9eaf-c8c900b3c677","Type":"ContainerDied","Data":"00ab22a7e4547d0e87d25b3f203110f7188f51b4ca37fce31ab9159e75204b98"}
Sep 07 11:27:59.299662 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:59.299626    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wggv8"
Sep 07 11:27:59.299825 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:27:59.299631    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ab22a7e4547d0e87d25b3f203110f7188f51b4ca37fce31ab9159e75204b98"
Sep 07 11:28:00.126713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:00.126670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:01.469857 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:01.469826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:03.744250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:03.744213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:03.844434 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:03.844392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:06.169414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.169380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:06.261656 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.261609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:06.262414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262389    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4c44n"]
Sep 07 11:28:06.262746 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262730    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerName="pull"
Sep 07 11:28:06.262829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262747    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerName="pull"
Sep 07 11:28:06.262829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerName="util"
Sep 07 11:28:06.262829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262777    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerName="util"
Sep 07 11:28:06.262829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262794    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerName="extract"
Sep 07 11:28:06.262829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262804    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerName="extract"
Sep 07 11:28:06.263061 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.262897    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4320e956-8c02-4206-9eaf-c8c900b3c677" containerName="extract"
Sep 07 11:28:06.266427 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.266406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.268800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.268782    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 07 11:28:06.269559 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.269542    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 07 11:28:06.269667 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.269547    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gcfjm\""
Sep 07 11:28:06.275062 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.275037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4c44n"]
Sep 07 11:28:06.360439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.360406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b464934-b45b-4a72-8484-dc38de599079-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4c44n\" (UID: \"4b464934-b45b-4a72-8484-dc38de599079\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.360619 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.360446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6v8\" (UniqueName: \"kubernetes.io/projected/4b464934-b45b-4a72-8484-dc38de599079-kube-api-access-4c6v8\") pod \"cert-manager-webhook-6486ffd997-4c44n\" (UID: \"4b464934-b45b-4a72-8484-dc38de599079\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.461100 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.461020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b464934-b45b-4a72-8484-dc38de599079-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4c44n\" (UID: \"4b464934-b45b-4a72-8484-dc38de599079\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.461100 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.461060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6v8\" (UniqueName: \"kubernetes.io/projected/4b464934-b45b-4a72-8484-dc38de599079-kube-api-access-4c6v8\") pod \"cert-manager-webhook-6486ffd997-4c44n\" (UID: \"4b464934-b45b-4a72-8484-dc38de599079\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.476048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.476016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b464934-b45b-4a72-8484-dc38de599079-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4c44n\" (UID: \"4b464934-b45b-4a72-8484-dc38de599079\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.479697 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.479672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6v8\" (UniqueName: \"kubernetes.io/projected/4b464934-b45b-4a72-8484-dc38de599079-kube-api-access-4c6v8\") pod \"cert-manager-webhook-6486ffd997-4c44n\" (UID: \"4b464934-b45b-4a72-8484-dc38de599079\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.565758 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.565714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:06.575734 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.575710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:06.667745 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.667712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:06.851036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.851003    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4c44n"]
Sep 07 11:28:06.853516 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:28:06.853489    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b464934_b45b_4a72_8484_dc38de599079.slice/crio-ebf009eb66850eaff9896eccf62bd36efd251ae3208eb0f3829c7cca9b3311ab WatchSource:0}: Error finding container ebf009eb66850eaff9896eccf62bd36efd251ae3208eb0f3829c7cca9b3311ab: Status 404 returned error can't find the container with id ebf009eb66850eaff9896eccf62bd36efd251ae3208eb0f3829c7cca9b3311ab
Sep 07 11:28:06.874567 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.874541    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"]
Sep 07 11:28:06.879317 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.879301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:06.881738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.881715    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 11:28:06.881838 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.881794    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 11:28:06.881928 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.881911    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm44s\""
Sep 07 11:28:06.887865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.887840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"]
Sep 07 11:28:06.966115 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.966080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:06.966271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.966139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:06.966271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:06.966191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lvg\" (UniqueName: \"kubernetes.io/projected/d8b18368-ffd0-4296-9712-012e50c103aa-kube-api-access-26lvg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.066966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.066930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.067115 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.066973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lvg\" (UniqueName: \"kubernetes.io/projected/d8b18368-ffd0-4296-9712-012e50c103aa-kube-api-access-26lvg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.067115 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.067062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.067393 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.067369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.067499 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.067413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.076266 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.076243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lvg\" (UniqueName: \"kubernetes.io/projected/d8b18368-ffd0-4296-9712-012e50c103aa-kube-api-access-26lvg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.189519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.189482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:07.333421 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.333382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n" event={"ID":"4b464934-b45b-4a72-8484-dc38de599079","Type":"ContainerStarted","Data":"ebf009eb66850eaff9896eccf62bd36efd251ae3208eb0f3829c7cca9b3311ab"}
Sep 07 11:28:07.461192 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:07.461169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"]
Sep 07 11:28:07.463288 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:28:07.463262    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b18368_ffd0_4296_9712_012e50c103aa.slice/crio-3b3f184a883db4588688a3a6e5280dcdee7a2e861577ebd22649ef9b98740d43 WatchSource:0}: Error finding container 3b3f184a883db4588688a3a6e5280dcdee7a2e861577ebd22649ef9b98740d43: Status 404 returned error can't find the container with id 3b3f184a883db4588688a3a6e5280dcdee7a2e861577ebd22649ef9b98740d43
Sep 07 11:28:08.341568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:08.341507    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8b18368-ffd0-4296-9712-012e50c103aa" containerID="9db0441a2a647d383739ff342026b77439cfc35b3d2b5a853bc00f414e67edf0" exitCode=0
Sep 07 11:28:08.342034 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:08.341631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh" event={"ID":"d8b18368-ffd0-4296-9712-012e50c103aa","Type":"ContainerDied","Data":"9db0441a2a647d383739ff342026b77439cfc35b3d2b5a853bc00f414e67edf0"}
Sep 07 11:28:08.342034 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:08.341683    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh" event={"ID":"d8b18368-ffd0-4296-9712-012e50c103aa","Type":"ContainerStarted","Data":"3b3f184a883db4588688a3a6e5280dcdee7a2e861577ebd22649ef9b98740d43"}
Sep 07 11:28:08.384236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:08.384205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:08.592965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:08.592888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:09.614754 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:09.614716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:10.354250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:10.354213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n" event={"ID":"4b464934-b45b-4a72-8484-dc38de599079","Type":"ContainerStarted","Data":"ec85a861646f5de8d4a1dd3ea1fdfdafac8dc326d8e8e9dab4813574c1fb27ae"}
Sep 07 11:28:10.354444 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:10.354317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:10.371980 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:10.371933    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n" podStartSLOduration=1.795501679 podStartE2EDuration="4.371914544s" totalImagesPullingTime="2.576412865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:28:10.371164061 +0000 UTC m=+321.846012245" watchObservedRunningTime="2026-09-07 11:28:10.371914544 +0000 UTC m=+321.846762694"
Sep 07 11:28:10.980136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:10.980109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:11.063640 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:11.063555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:11.360152 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:11.360063    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8b18368-ffd0-4296-9712-012e50c103aa" containerID="e132ad08491f37a296de2b696db861cafe3839fba4f8a5d0bee009f2fa739017" exitCode=0
Sep 07 11:28:11.360152 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:11.360136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh" event={"ID":"d8b18368-ffd0-4296-9712-012e50c103aa","Type":"ContainerDied","Data":"e132ad08491f37a296de2b696db861cafe3839fba4f8a5d0bee009f2fa739017"}
Sep 07 11:28:12.366997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:12.366960    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8b18368-ffd0-4296-9712-012e50c103aa" containerID="6634a585ef729e769e289d8bd786583ebfe7252d463ffc6ce81206bc5491f9a2" exitCode=0
Sep 07 11:28:12.367375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:12.367032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh" event={"ID":"d8b18368-ffd0-4296-9712-012e50c103aa","Type":"ContainerDied","Data":"6634a585ef729e769e289d8bd786583ebfe7252d463ffc6ce81206bc5491f9a2"}
Sep 07 11:28:13.494653 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.494626    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:13.525781 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.525745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-util\") pod \"d8b18368-ffd0-4296-9712-012e50c103aa\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") "
Sep 07 11:28:13.525936 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.525871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-bundle\") pod \"d8b18368-ffd0-4296-9712-012e50c103aa\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") "
Sep 07 11:28:13.525936 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.525910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26lvg\" (UniqueName: \"kubernetes.io/projected/d8b18368-ffd0-4296-9712-012e50c103aa-kube-api-access-26lvg\") pod \"d8b18368-ffd0-4296-9712-012e50c103aa\" (UID: \"d8b18368-ffd0-4296-9712-012e50c103aa\") "
Sep 07 11:28:13.526254 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.526228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-bundle" (OuterVolumeSpecName: "bundle") pod "d8b18368-ffd0-4296-9712-012e50c103aa" (UID: "d8b18368-ffd0-4296-9712-012e50c103aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:28:13.527953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.527928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b18368-ffd0-4296-9712-012e50c103aa-kube-api-access-26lvg" (OuterVolumeSpecName: "kube-api-access-26lvg") pod "d8b18368-ffd0-4296-9712-012e50c103aa" (UID: "d8b18368-ffd0-4296-9712-012e50c103aa"). InnerVolumeSpecName "kube-api-access-26lvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:28:13.530302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.530274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-util" (OuterVolumeSpecName: "util") pod "d8b18368-ffd0-4296-9712-012e50c103aa" (UID: "d8b18368-ffd0-4296-9712-012e50c103aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:28:13.626800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.626734    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:13.626800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.626760    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8b18368-ffd0-4296-9712-012e50c103aa-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:13.626800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:13.626769    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lvg\" (UniqueName: \"kubernetes.io/projected/d8b18368-ffd0-4296-9712-012e50c103aa-kube-api-access-26lvg\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:14.377297 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:14.377267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh"
Sep 07 11:28:14.377297 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:14.377280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fjkh" event={"ID":"d8b18368-ffd0-4296-9712-012e50c103aa","Type":"ContainerDied","Data":"3b3f184a883db4588688a3a6e5280dcdee7a2e861577ebd22649ef9b98740d43"}
Sep 07 11:28:14.377529 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:14.377322    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b3f184a883db4588688a3a6e5280dcdee7a2e861577ebd22649ef9b98740d43"
Sep 07 11:28:16.362473 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.362425    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4c44n"
Sep 07 11:28:16.362871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.362560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:16.452030 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.451997    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-slbvv"]
Sep 07 11:28:16.452321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.452308    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b18368-ffd0-4296-9712-012e50c103aa" containerName="util"
Sep 07 11:28:16.452362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.452323    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b18368-ffd0-4296-9712-012e50c103aa" containerName="util"
Sep 07 11:28:16.452362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.452334    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b18368-ffd0-4296-9712-012e50c103aa" containerName="pull"
Sep 07 11:28:16.452362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.452339    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b18368-ffd0-4296-9712-012e50c103aa" containerName="pull"
Sep 07 11:28:16.452362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.452350    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b18368-ffd0-4296-9712-012e50c103aa" containerName="extract"
Sep 07 11:28:16.452362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.452356    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b18368-ffd0-4296-9712-012e50c103aa" containerName="extract"
Sep 07 11:28:16.452537 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.452433    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b18368-ffd0-4296-9712-012e50c103aa" containerName="extract"
Sep 07 11:28:16.456423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.456408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.458680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.458663    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8l4dt\""
Sep 07 11:28:16.468315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.468270    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-slbvv"]
Sep 07 11:28:16.489634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.489604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:16.545933 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.545899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6ef4a6-ce54-451f-a968-f01ec06f72f0-bound-sa-token\") pod \"cert-manager-55b4d954d5-slbvv\" (UID: \"ab6ef4a6-ce54-451f-a968-f01ec06f72f0\") " pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.546079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.545958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6p4s\" (UniqueName: \"kubernetes.io/projected/ab6ef4a6-ce54-451f-a968-f01ec06f72f0-kube-api-access-t6p4s\") pod \"cert-manager-55b4d954d5-slbvv\" (UID: \"ab6ef4a6-ce54-451f-a968-f01ec06f72f0\") " pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.647130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.647050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6ef4a6-ce54-451f-a968-f01ec06f72f0-bound-sa-token\") pod \"cert-manager-55b4d954d5-slbvv\" (UID: \"ab6ef4a6-ce54-451f-a968-f01ec06f72f0\") " pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.647130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.647111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6p4s\" (UniqueName: \"kubernetes.io/projected/ab6ef4a6-ce54-451f-a968-f01ec06f72f0-kube-api-access-t6p4s\") pod \"cert-manager-55b4d954d5-slbvv\" (UID: \"ab6ef4a6-ce54-451f-a968-f01ec06f72f0\") " pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.656162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.656137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6ef4a6-ce54-451f-a968-f01ec06f72f0-bound-sa-token\") pod \"cert-manager-55b4d954d5-slbvv\" (UID: \"ab6ef4a6-ce54-451f-a968-f01ec06f72f0\") " pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.656162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.656151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6p4s\" (UniqueName: \"kubernetes.io/projected/ab6ef4a6-ce54-451f-a968-f01ec06f72f0-kube-api-access-t6p4s\") pod \"cert-manager-55b4d954d5-slbvv\" (UID: \"ab6ef4a6-ce54-451f-a968-f01ec06f72f0\") " pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.765293 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.765254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-slbvv"
Sep 07 11:28:16.994550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:16.994454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:17.039126 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:17.039087    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-slbvv"]
Sep 07 11:28:17.041786 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:28:17.041758    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab6ef4a6_ce54_451f_a968_f01ec06f72f0.slice/crio-82f1a77d851b4c15f41bef6106d460f3c9253f742347cfa4795f34413b1348a1 WatchSource:0}: Error finding container 82f1a77d851b4c15f41bef6106d460f3c9253f742347cfa4795f34413b1348a1: Status 404 returned error can't find the container with id 82f1a77d851b4c15f41bef6106d460f3c9253f742347cfa4795f34413b1348a1
Sep 07 11:28:17.131272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:17.131249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:17.391608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:17.391575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-slbvv" event={"ID":"ab6ef4a6-ce54-451f-a968-f01ec06f72f0","Type":"ContainerStarted","Data":"2945ec7bffbbe9cb162d0a421b3fcbcca1f3055f2ba88b5a54ad5567254aa0dc"}
Sep 07 11:28:17.391991 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:17.391616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-slbvv" event={"ID":"ab6ef4a6-ce54-451f-a968-f01ec06f72f0","Type":"ContainerStarted","Data":"82f1a77d851b4c15f41bef6106d460f3c9253f742347cfa4795f34413b1348a1"}
Sep 07 11:28:17.408553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:17.408498    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-slbvv" podStartSLOduration=1.4084803350000001 podStartE2EDuration="1.408480335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:28:17.40681071 +0000 UTC m=+328.881658860" watchObservedRunningTime="2026-09-07 11:28:17.408480335 +0000 UTC m=+328.883328478"
Sep 07 11:28:17.969878 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:17.969845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:18.067212 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:18.067182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:23.781410 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:23.781377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:25.373025 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:25.372990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:26.799736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:26.799697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:28.151687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.151656    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"]
Sep 07 11:28:28.194705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.194676    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"]
Sep 07 11:28:28.194842 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.194772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.197725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.197701    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 11:28:28.197725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.197717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 11:28:28.198639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.198621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm44s\""
Sep 07 11:28:28.228362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.228336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgc4d\" (UniqueName: \"kubernetes.io/projected/5a9182d6-4402-43bb-a458-13af4224eb76-kube-api-access-tgc4d\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.228457 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.228376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.228457 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.228394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.329747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.329719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgc4d\" (UniqueName: \"kubernetes.io/projected/5a9182d6-4402-43bb-a458-13af4224eb76-kube-api-access-tgc4d\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.329865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.329773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.329865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.329805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.330203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.330182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.330266 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.330200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.337491 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.337452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgc4d\" (UniqueName: \"kubernetes.io/projected/5a9182d6-4402-43bb-a458-13af4224eb76-kube-api-access-tgc4d\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.504039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.503966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:28.772073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:28.772048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"]
Sep 07 11:28:28.774205 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:28:28.774176    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9182d6_4402_43bb_a458_13af4224eb76.slice/crio-610981b3170f2feccb83b6a301ef069315934d3a8858b02d433713862290aa2a WatchSource:0}: Error finding container 610981b3170f2feccb83b6a301ef069315934d3a8858b02d433713862290aa2a: Status 404 returned error can't find the container with id 610981b3170f2feccb83b6a301ef069315934d3a8858b02d433713862290aa2a
Sep 07 11:28:29.450796 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:29.450762    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a9182d6-4402-43bb-a458-13af4224eb76" containerID="820f0879e451f193df763f346e9b3abab6353b906b868c6a38fb2f35414c6680" exitCode=0
Sep 07 11:28:29.451125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:29.450807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs" event={"ID":"5a9182d6-4402-43bb-a458-13af4224eb76","Type":"ContainerDied","Data":"820f0879e451f193df763f346e9b3abab6353b906b868c6a38fb2f35414c6680"}
Sep 07 11:28:29.451125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:29.450841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs" event={"ID":"5a9182d6-4402-43bb-a458-13af4224eb76","Type":"ContainerStarted","Data":"610981b3170f2feccb83b6a301ef069315934d3a8858b02d433713862290aa2a"}
Sep 07 11:28:30.457337 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:30.457304    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a9182d6-4402-43bb-a458-13af4224eb76" containerID="575d9d46b4de4e45485a53755c32d50155aedf32f1dc4af357608a5da09a94b9" exitCode=0
Sep 07 11:28:30.457727 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:30.457353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs" event={"ID":"5a9182d6-4402-43bb-a458-13af4224eb76","Type":"ContainerDied","Data":"575d9d46b4de4e45485a53755c32d50155aedf32f1dc4af357608a5da09a94b9"}
Sep 07 11:28:31.464201 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:31.464168    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a9182d6-4402-43bb-a458-13af4224eb76" containerID="d7f20d6fe60faa66df2468b369862110f0cbf3e8d41881998734f5d523bff0db" exitCode=0
Sep 07 11:28:31.464582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:31.464252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs" event={"ID":"5a9182d6-4402-43bb-a458-13af4224eb76","Type":"ContainerDied","Data":"d7f20d6fe60faa66df2468b369862110f0cbf3e8d41881998734f5d523bff0db"}
Sep 07 11:28:32.586102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.586074    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:32.664027 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.663997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-util\") pod \"5a9182d6-4402-43bb-a458-13af4224eb76\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") "
Sep 07 11:28:32.664174 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.664048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-bundle\") pod \"5a9182d6-4402-43bb-a458-13af4224eb76\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") "
Sep 07 11:28:32.664174 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.664081    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgc4d\" (UniqueName: \"kubernetes.io/projected/5a9182d6-4402-43bb-a458-13af4224eb76-kube-api-access-tgc4d\") pod \"5a9182d6-4402-43bb-a458-13af4224eb76\" (UID: \"5a9182d6-4402-43bb-a458-13af4224eb76\") "
Sep 07 11:28:32.664967 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.664935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-bundle" (OuterVolumeSpecName: "bundle") pod "5a9182d6-4402-43bb-a458-13af4224eb76" (UID: "5a9182d6-4402-43bb-a458-13af4224eb76"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:28:32.666182 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.666153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9182d6-4402-43bb-a458-13af4224eb76-kube-api-access-tgc4d" (OuterVolumeSpecName: "kube-api-access-tgc4d") pod "5a9182d6-4402-43bb-a458-13af4224eb76" (UID: "5a9182d6-4402-43bb-a458-13af4224eb76"). InnerVolumeSpecName "kube-api-access-tgc4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:28:32.670150 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.670127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-util" (OuterVolumeSpecName: "util") pod "5a9182d6-4402-43bb-a458-13af4224eb76" (UID: "5a9182d6-4402-43bb-a458-13af4224eb76"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:28:32.764890 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.764833    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:32.764890 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.764854    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a9182d6-4402-43bb-a458-13af4224eb76-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:32.764890 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:32.764863    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgc4d\" (UniqueName: \"kubernetes.io/projected/5a9182d6-4402-43bb-a458-13af4224eb76-kube-api-access-tgc4d\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:33.474238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:33.474201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs" event={"ID":"5a9182d6-4402-43bb-a458-13af4224eb76","Type":"ContainerDied","Data":"610981b3170f2feccb83b6a301ef069315934d3a8858b02d433713862290aa2a"}
Sep 07 11:28:33.474238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:33.474225    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9zzs"
Sep 07 11:28:33.474438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:33.474273    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610981b3170f2feccb83b6a301ef069315934d3a8858b02d433713862290aa2a"
Sep 07 11:28:43.103317 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.103237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:43.162835 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.162805    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"]
Sep 07 11:28:43.163151 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.163138    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9182d6-4402-43bb-a458-13af4224eb76" containerName="util"
Sep 07 11:28:43.163195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.163153    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9182d6-4402-43bb-a458-13af4224eb76" containerName="util"
Sep 07 11:28:43.163195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.163163    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9182d6-4402-43bb-a458-13af4224eb76" containerName="extract"
Sep 07 11:28:43.163195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.163168    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9182d6-4402-43bb-a458-13af4224eb76" containerName="extract"
Sep 07 11:28:43.163195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.163178    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9182d6-4402-43bb-a458-13af4224eb76" containerName="pull"
Sep 07 11:28:43.163195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.163184    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9182d6-4402-43bb-a458-13af4224eb76" containerName="pull"
Sep 07 11:28:43.163342 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.163244    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9182d6-4402-43bb-a458-13af4224eb76" containerName="extract"
Sep 07 11:28:43.166212 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.166196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.168679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.168659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 11:28:43.169729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.169699    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 11:28:43.169729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.169716    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 11:28:43.169729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.169731    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:28:43.169935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.169740    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8z5pf\""
Sep 07 11:28:43.169935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.169796    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 11:28:43.175907 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.175888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"]
Sep 07 11:28:43.211805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.211767    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:43.249406 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.249370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-metrics-cert\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.249583 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.249415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-manager-config\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.249583 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.249486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-cert\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.249687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.249572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgrm\" (UniqueName: \"kubernetes.io/projected/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-kube-api-access-4mgrm\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.350483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.350434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-manager-config\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.350673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.350522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-cert\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.350673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.350557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgrm\" (UniqueName: \"kubernetes.io/projected/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-kube-api-access-4mgrm\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.350673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.350590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-metrics-cert\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.351192 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.351165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-manager-config\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.353190 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.353166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-cert\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.353245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.353166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-metrics-cert\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.358501 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.358435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgrm\" (UniqueName: \"kubernetes.io/projected/bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2-kube-api-access-4mgrm\") pod \"lws-controller-manager-79b5d8b54d-55ptn\" (UID: \"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2\") " pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.395801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.395770    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"]
Sep 07 11:28:43.399390 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.399375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.401876 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.401847    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 11:28:43.401971 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.401853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 11:28:43.401971 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.401890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm44s\""
Sep 07 11:28:43.414445 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.414418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"]
Sep 07 11:28:43.451867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.451839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.451979 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.451890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr26w\" (UniqueName: \"kubernetes.io/projected/2419c81b-cd93-4c07-acfe-e6a922a8db99-kube-api-access-nr26w\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.452029 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.451990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.475852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.475823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:43.553031 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.552997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.553167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.553048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr26w\" (UniqueName: \"kubernetes.io/projected/2419c81b-cd93-4c07-acfe-e6a922a8db99-kube-api-access-nr26w\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.553237 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.553187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.553531 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.553502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.553531 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.553518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.561799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.561771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr26w\" (UniqueName: \"kubernetes.io/projected/2419c81b-cd93-4c07-acfe-e6a922a8db99-kube-api-access-nr26w\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.708390 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.708363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:43.761231 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.761101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"]
Sep 07 11:28:43.765964 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:28:43.765937    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb334a9e_9a3d_4ab5_bc42_d39dc153b4d2.slice/crio-00a08e153fab49d13ea844156ead8db082af7c59e75d2fb7cc6ba913b39a257e WatchSource:0}: Error finding container 00a08e153fab49d13ea844156ead8db082af7c59e75d2fb7cc6ba913b39a257e: Status 404 returned error can't find the container with id 00a08e153fab49d13ea844156ead8db082af7c59e75d2fb7cc6ba913b39a257e
Sep 07 11:28:43.981301 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:43.981279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"]
Sep 07 11:28:43.983205 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:28:43.983180    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2419c81b_cd93_4c07_acfe_e6a922a8db99.slice/crio-4f67fb5a25a9949296bc093d4500d864be2dd41af3219e902088a5d03c38525c WatchSource:0}: Error finding container 4f67fb5a25a9949296bc093d4500d864be2dd41af3219e902088a5d03c38525c: Status 404 returned error can't find the container with id 4f67fb5a25a9949296bc093d4500d864be2dd41af3219e902088a5d03c38525c
Sep 07 11:28:44.532970 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:44.532936    2578 generic.go:356] "Generic (PLEG): container finished" podID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerID="fc7a9343268654a889c7e7474a7e49ff700e1a16d5eec74880720b8bf801165a" exitCode=0
Sep 07 11:28:44.533515 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:44.533028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4" event={"ID":"2419c81b-cd93-4c07-acfe-e6a922a8db99","Type":"ContainerDied","Data":"fc7a9343268654a889c7e7474a7e49ff700e1a16d5eec74880720b8bf801165a"}
Sep 07 11:28:44.533515 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:44.533118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4" event={"ID":"2419c81b-cd93-4c07-acfe-e6a922a8db99","Type":"ContainerStarted","Data":"4f67fb5a25a9949296bc093d4500d864be2dd41af3219e902088a5d03c38525c"}
Sep 07 11:28:44.534589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:44.534563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn" event={"ID":"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2","Type":"ContainerStarted","Data":"00a08e153fab49d13ea844156ead8db082af7c59e75d2fb7cc6ba913b39a257e"}
Sep 07 11:28:44.780648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:44.780610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:44.880009 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:44.879962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:45.376968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.376939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:45.541480 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.541374    2578 generic.go:356] "Generic (PLEG): container finished" podID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerID="9a47a984039c91401b0445fac569d2b40eb6fd19060a74215bec022dcae59122" exitCode=0
Sep 07 11:28:45.541923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.541490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4" event={"ID":"2419c81b-cd93-4c07-acfe-e6a922a8db99","Type":"ContainerDied","Data":"9a47a984039c91401b0445fac569d2b40eb6fd19060a74215bec022dcae59122"}
Sep 07 11:28:45.543549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.543454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn" event={"ID":"bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2","Type":"ContainerStarted","Data":"d35b4b178623125db76835a420cefbc904bfb8682384b960343d97ef08b90cba"}
Sep 07 11:28:45.543673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.543568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:28:45.590541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.590496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn" podStartSLOduration=1.315946732 podStartE2EDuration="2.590480656s" totalImagesPullingTime="1.274533924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:28:45.588502077 +0000 UTC m=+357.063350227" watchObservedRunningTime="2026-09-07 11:28:45.590480656 +0000 UTC m=+357.065328854"
Sep 07 11:28:45.601566 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.601543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:45.663390 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.663361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:45.761418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.761387    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"]
Sep 07 11:28:45.764883 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.764862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.767453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.767427    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g8bwf\""
Sep 07 11:28:45.768315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.768298    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 11:28:45.768914 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.768892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 11:28:45.769028 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.768915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 11:28:45.769299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.769274    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 11:28:45.788811 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.788786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"]
Sep 07 11:28:45.872826 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.872792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhr7\" (UniqueName: \"kubernetes.io/projected/1836dbd9-8f98-47ed-853a-2b660fa48a89-kube-api-access-9zhr7\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.872962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.872831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1836dbd9-8f98-47ed-853a-2b660fa48a89-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.872962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.872904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1836dbd9-8f98-47ed-853a-2b660fa48a89-webhook-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.872962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.872936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1836dbd9-8f98-47ed-853a-2b660fa48a89-tmp\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.873082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.872961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1836dbd9-8f98-47ed-853a-2b660fa48a89-manifests\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.896040 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.896010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:45.973557 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.973523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1836dbd9-8f98-47ed-853a-2b660fa48a89-webhook-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.973675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.973563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1836dbd9-8f98-47ed-853a-2b660fa48a89-tmp\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.973675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.973581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1836dbd9-8f98-47ed-853a-2b660fa48a89-manifests\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.973675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.973650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhr7\" (UniqueName: \"kubernetes.io/projected/1836dbd9-8f98-47ed-853a-2b660fa48a89-kube-api-access-9zhr7\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.973806 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.973676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1836dbd9-8f98-47ed-853a-2b660fa48a89-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.974084 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.974061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1836dbd9-8f98-47ed-853a-2b660fa48a89-manifests\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.975911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.975890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1836dbd9-8f98-47ed-853a-2b660fa48a89-tmp\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.976126 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.976103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1836dbd9-8f98-47ed-853a-2b660fa48a89-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.976203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.976189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1836dbd9-8f98-47ed-853a-2b660fa48a89-webhook-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.981013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.980987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhr7\" (UniqueName: \"kubernetes.io/projected/1836dbd9-8f98-47ed-853a-2b660fa48a89-kube-api-access-9zhr7\") pod \"opendatahub-operator-controller-manager-f479d7474-g5gb8\" (UID: \"1836dbd9-8f98-47ed-853a-2b660fa48a89\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:45.988692 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:45.988660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:46.074814 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.074787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:46.089166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.089141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:46.189362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.189332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:46.295624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.295593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:46.350831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.350809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"]
Sep 07 11:28:46.353050 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:28:46.353027    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1836dbd9_8f98_47ed_853a_2b660fa48a89.slice/crio-f15c80256218de5a414d9534b3110592dc17b06da07f3bf2a58cd500ff8db55f WatchSource:0}: Error finding container f15c80256218de5a414d9534b3110592dc17b06da07f3bf2a58cd500ff8db55f: Status 404 returned error can't find the container with id f15c80256218de5a414d9534b3110592dc17b06da07f3bf2a58cd500ff8db55f
Sep 07 11:28:46.359170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.359146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:28:46.549189 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.549103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8" event={"ID":"1836dbd9-8f98-47ed-853a-2b660fa48a89","Type":"ContainerStarted","Data":"f15c80256218de5a414d9534b3110592dc17b06da07f3bf2a58cd500ff8db55f"}
Sep 07 11:28:46.551288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.551263    2578 generic.go:356] "Generic (PLEG): container finished" podID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerID="585e2e22dcee33175630e76edcb23d8ac8c42a844c33422f3c3ee794fb3534fb" exitCode=0
Sep 07 11:28:46.551408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:46.551345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4" event={"ID":"2419c81b-cd93-4c07-acfe-e6a922a8db99","Type":"ContainerDied","Data":"585e2e22dcee33175630e76edcb23d8ac8c42a844c33422f3c3ee794fb3534fb"}
Sep 07 11:28:47.722663 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.722632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:47.791388 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.791356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr26w\" (UniqueName: \"kubernetes.io/projected/2419c81b-cd93-4c07-acfe-e6a922a8db99-kube-api-access-nr26w\") pod \"2419c81b-cd93-4c07-acfe-e6a922a8db99\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") "
Sep 07 11:28:47.791582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.791419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-bundle\") pod \"2419c81b-cd93-4c07-acfe-e6a922a8db99\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") "
Sep 07 11:28:47.791582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.791452    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-util\") pod \"2419c81b-cd93-4c07-acfe-e6a922a8db99\" (UID: \"2419c81b-cd93-4c07-acfe-e6a922a8db99\") "
Sep 07 11:28:47.792593 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.792562    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-bundle" (OuterVolumeSpecName: "bundle") pod "2419c81b-cd93-4c07-acfe-e6a922a8db99" (UID: "2419c81b-cd93-4c07-acfe-e6a922a8db99"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:28:47.794736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.794705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2419c81b-cd93-4c07-acfe-e6a922a8db99-kube-api-access-nr26w" (OuterVolumeSpecName: "kube-api-access-nr26w") pod "2419c81b-cd93-4c07-acfe-e6a922a8db99" (UID: "2419c81b-cd93-4c07-acfe-e6a922a8db99"). InnerVolumeSpecName "kube-api-access-nr26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:28:47.800115 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.800090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-util" (OuterVolumeSpecName: "util") pod "2419c81b-cd93-4c07-acfe-e6a922a8db99" (UID: "2419c81b-cd93-4c07-acfe-e6a922a8db99"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:28:47.892728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.892697    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:47.892728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.892727    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2419c81b-cd93-4c07-acfe-e6a922a8db99-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:47.892728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:47.892740    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr26w\" (UniqueName: \"kubernetes.io/projected/2419c81b-cd93-4c07-acfe-e6a922a8db99-kube-api-access-nr26w\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:28:48.564889 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:48.564848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4" event={"ID":"2419c81b-cd93-4c07-acfe-e6a922a8db99","Type":"ContainerDied","Data":"4f67fb5a25a9949296bc093d4500d864be2dd41af3219e902088a5d03c38525c"}
Sep 07 11:28:48.564889 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:48.564882    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0w5pt4"
Sep 07 11:28:48.565186 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:48.564901    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f67fb5a25a9949296bc093d4500d864be2dd41af3219e902088a5d03c38525c"
Sep 07 11:28:49.572195 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:49.572156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8" event={"ID":"1836dbd9-8f98-47ed-853a-2b660fa48a89","Type":"ContainerStarted","Data":"0dc677a5873361f33e1d6e4983e9f189c47416e887f4de4720086ed086e1f27c"}
Sep 07 11:28:51.583318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:51.583281    2578 generic.go:356] "Generic (PLEG): container finished" podID="1836dbd9-8f98-47ed-853a-2b660fa48a89" containerID="0dc677a5873361f33e1d6e4983e9f189c47416e887f4de4720086ed086e1f27c" exitCode=0
Sep 07 11:28:51.583697 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:51.583354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8" event={"ID":"1836dbd9-8f98-47ed-853a-2b660fa48a89","Type":"ContainerDied","Data":"0dc677a5873361f33e1d6e4983e9f189c47416e887f4de4720086ed086e1f27c"}
Sep 07 11:28:52.589671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:52.589631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8" event={"ID":"1836dbd9-8f98-47ed-853a-2b660fa48a89","Type":"ContainerStarted","Data":"3b21f62a3f2b6b3caea29e48faf32d7fde0bec3810f7e488e0ac521ec8bce547"}
Sep 07 11:28:52.590147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:52.589924    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:28:52.612959 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:52.612912    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8" podStartSLOduration=3.148205804 podStartE2EDuration="7.612896493s" totalImagesPullingTime="3.4296903s" totalInitContainerRuntime="1.035000389s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:28:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:28:52.609938371 +0000 UTC m=+364.084786520" watchObservedRunningTime="2026-09-07 11:28:52.612896493 +0000 UTC m=+364.087744705"
Sep 07 11:28:56.553786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:28:56.553746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79b5d8b54d-55ptn"
Sep 07 11:29:03.596244 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:03.596212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-g5gb8"
Sep 07 11:29:06.385256 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.385228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:06.493599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.493567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:06.588755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.588721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:06.625167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625136    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"]
Sep 07 11:29:06.625535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625518    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerName="util"
Sep 07 11:29:06.625535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625536    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerName="util"
Sep 07 11:29:06.625641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625545    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerName="extract"
Sep 07 11:29:06.625641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerName="extract"
Sep 07 11:29:06.625641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625558    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerName="pull"
Sep 07 11:29:06.625641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625564    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerName="pull"
Sep 07 11:29:06.625641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.625615    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2419c81b-cd93-4c07-acfe-e6a922a8db99" containerName="extract"
Sep 07 11:29:06.633644 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.633620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.636349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.636294    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 11:29:06.636349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.636309    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 11:29:06.637080 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.637062    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 11:29:06.637169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.637067    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-72d9d\""
Sep 07 11:29:06.637169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.637070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 11:29:06.641450 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.641430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"]
Sep 07 11:29:06.742895 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.742864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tmp\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.743049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.742904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.743049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.742937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtp7j\" (UniqueName: \"kubernetes.io/projected/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-kube-api-access-rtp7j\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.843851 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.843814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tmp\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.843990 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.843870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.843990 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.843920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtp7j\" (UniqueName: \"kubernetes.io/projected/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-kube-api-access-rtp7j\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.844095 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:29:06.843993    2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 07 11:29:06.844095 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:29:06.844060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tls-certs podName:33e2bc5b-66b3-4090-865d-d4ea6e08e32c nodeName:}" failed. No retries permitted until 2026-09-07 11:29:07.344045287 +0000 UTC m=+378.818893414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tls-certs") pod "kube-auth-proxy-5586fdf7b-jxnn7" (UID: "33e2bc5b-66b3-4090-865d-d4ea6e08e32c") : secret "kube-auth-proxy-tls" not found
Sep 07 11:29:06.846161 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.846139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tmp\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.852184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.852162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtp7j\" (UniqueName: \"kubernetes.io/projected/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-kube-api-access-rtp7j\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:06.912390 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:06.912318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:07.347314 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:07.347230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:07.349678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:07.349655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e2bc5b-66b3-4090-865d-d4ea6e08e32c-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-jxnn7\" (UID: \"33e2bc5b-66b3-4090-865d-d4ea6e08e32c\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:07.544688 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:07.544654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"
Sep 07 11:29:07.820413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:07.820366    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7"]
Sep 07 11:29:07.822648 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:29:07.822618    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e2bc5b_66b3_4090_865d_d4ea6e08e32c.slice/crio-ccebe3f035bd2c811180b1340830398cbc9e8aca1d3a2d099686edd6cec9b06d WatchSource:0}: Error finding container ccebe3f035bd2c811180b1340830398cbc9e8aca1d3a2d099686edd6cec9b06d: Status 404 returned error can't find the container with id ccebe3f035bd2c811180b1340830398cbc9e8aca1d3a2d099686edd6cec9b06d
Sep 07 11:29:07.916716 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:07.916687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:07.975326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:07.975290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:08.671176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:08.671092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7" event={"ID":"33e2bc5b-66b3-4090-865d-d4ea6e08e32c","Type":"ContainerStarted","Data":"ccebe3f035bd2c811180b1340830398cbc9e8aca1d3a2d099686edd6cec9b06d"}
Sep 07 11:29:08.903157 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:08.903115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:10.683126 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:10.683092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7" event={"ID":"33e2bc5b-66b3-4090-865d-d4ea6e08e32c","Type":"ContainerStarted","Data":"460ece2bbfcecd28fd7f61b1ebf5010cf1e84fc1b484495f23cfc7d1afe10027"}
Sep 07 11:29:10.700681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:10.700630    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-jxnn7" podStartSLOduration=2.026798693 podStartE2EDuration="4.700616334s" totalImagesPullingTime="2.673817641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:29:10.699544336 +0000 UTC m=+382.174392498" watchObservedRunningTime="2026-09-07 11:29:10.700616334 +0000 UTC m=+382.175464513"
Sep 07 11:29:13.181749 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:13.181711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:13.241875 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:13.241846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:13.853283 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:13.853253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:13.957694 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:13.957662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:14.053435 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:14.053404    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:17.304410 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:17.304371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:17.405163 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:17.405133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:17.507826 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:17.507796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:17.720097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:17.720066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:18.181506 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:18.181446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:27.670798 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:27.670728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:27.755211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:27.755165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:27.993533 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:27.993438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:28.493688 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:28.493658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:28.957250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:28.957213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:29.050098 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:29.050068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:29.150283 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:29.150256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:40.817668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:40.817614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:40.881407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:40.881353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:51.208574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:51.208536    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d8f5d6cd9-wn77d"]
Sep 07 11:29:51.317200 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:51.317155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:51.418402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:51.418365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:51.490407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:51.490319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:59.166208 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:59.166172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:29:59.232592 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:29:59.232561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:01.147363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:01.147322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:01.248599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:01.248548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:16.231479 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.231353    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d8f5d6cd9-wn77d" podUID="4d92c80e-0c04-4553-9e01-08f40da9edf3" containerName="console" containerID="cri-o://4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73" gracePeriod=15
Sep 07 11:30:16.473014 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.472987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:30:16.551161 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnh69\" (UniqueName: \"kubernetes.io/projected/4d92c80e-0c04-4553-9e01-08f40da9edf3-kube-api-access-hnh69\") pod \"4d92c80e-0c04-4553-9e01-08f40da9edf3\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") "
Sep 07 11:30:16.551297 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-trusted-ca-bundle\") pod \"4d92c80e-0c04-4553-9e01-08f40da9edf3\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") "
Sep 07 11:30:16.551297 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-oauth-serving-cert\") pod \"4d92c80e-0c04-4553-9e01-08f40da9edf3\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") "
Sep 07 11:30:16.551297 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-serving-cert\") pod \"4d92c80e-0c04-4553-9e01-08f40da9edf3\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") "
Sep 07 11:30:16.551449 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-oauth-config\") pod \"4d92c80e-0c04-4553-9e01-08f40da9edf3\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") "
Sep 07 11:30:16.551449 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-service-ca\") pod \"4d92c80e-0c04-4553-9e01-08f40da9edf3\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") "
Sep 07 11:30:16.551449 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-config\") pod \"4d92c80e-0c04-4553-9e01-08f40da9edf3\" (UID: \"4d92c80e-0c04-4553-9e01-08f40da9edf3\") "
Sep 07 11:30:16.551727 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4d92c80e-0c04-4553-9e01-08f40da9edf3" (UID: "4d92c80e-0c04-4553-9e01-08f40da9edf3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:30:16.551876 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d92c80e-0c04-4553-9e01-08f40da9edf3" (UID: "4d92c80e-0c04-4553-9e01-08f40da9edf3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:30:16.552026 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.551833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d92c80e-0c04-4553-9e01-08f40da9edf3" (UID: "4d92c80e-0c04-4553-9e01-08f40da9edf3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:30:16.552162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.552106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-config" (OuterVolumeSpecName: "console-config") pod "4d92c80e-0c04-4553-9e01-08f40da9edf3" (UID: "4d92c80e-0c04-4553-9e01-08f40da9edf3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:30:16.553383 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.553354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d92c80e-0c04-4553-9e01-08f40da9edf3-kube-api-access-hnh69" (OuterVolumeSpecName: "kube-api-access-hnh69") pod "4d92c80e-0c04-4553-9e01-08f40da9edf3" (UID: "4d92c80e-0c04-4553-9e01-08f40da9edf3"). InnerVolumeSpecName "kube-api-access-hnh69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:30:16.553508 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.553444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d92c80e-0c04-4553-9e01-08f40da9edf3" (UID: "4d92c80e-0c04-4553-9e01-08f40da9edf3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:30:16.553572 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.553538    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d92c80e-0c04-4553-9e01-08f40da9edf3" (UID: "4d92c80e-0c04-4553-9e01-08f40da9edf3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:30:16.652614 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.652591    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-trusted-ca-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:16.652614 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.652611    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-oauth-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:16.652740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.652621    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:16.652740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.652629    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-oauth-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:16.652740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.652638    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:16.652740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.652646    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d92c80e-0c04-4553-9e01-08f40da9edf3-console-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:16.652740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:16.652654    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnh69\" (UniqueName: \"kubernetes.io/projected/4d92c80e-0c04-4553-9e01-08f40da9edf3-kube-api-access-hnh69\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:17.008987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.008950    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d92c80e-0c04-4553-9e01-08f40da9edf3" containerID="4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73" exitCode=0
Sep 07 11:30:17.009132 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.008998    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8f5d6cd9-wn77d"
Sep 07 11:30:17.009132 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.009030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8f5d6cd9-wn77d" event={"ID":"4d92c80e-0c04-4553-9e01-08f40da9edf3","Type":"ContainerDied","Data":"4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73"}
Sep 07 11:30:17.009132 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.009081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8f5d6cd9-wn77d" event={"ID":"4d92c80e-0c04-4553-9e01-08f40da9edf3","Type":"ContainerDied","Data":"be41d6783bf642d31bc4575ed48747ac5608f68425897c63b6d315db82ce6cb4"}
Sep 07 11:30:17.009132 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.009101    2578 scope.go:117] "RemoveContainer" containerID="4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73"
Sep 07 11:30:17.017576 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.017560    2578 scope.go:117] "RemoveContainer" containerID="4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73"
Sep 07 11:30:17.017865 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:30:17.017841    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73\": container with ID starting with 4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73 not found: ID does not exist" containerID="4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73"
Sep 07 11:30:17.017959 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.017870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73"} err="failed to get container status \"4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73\": rpc error: code = NotFound desc = could not find container \"4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73\": container with ID starting with 4ad980b7094776f3f8441caf9a9fb49934e58486d401dbab8fd2347d3959bc73 not found: ID does not exist"
Sep 07 11:30:17.036027 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.036003    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d8f5d6cd9-wn77d"]
Sep 07 11:30:17.044011 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:17.043987    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d8f5d6cd9-wn77d"]
Sep 07 11:30:18.681923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:18.681884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:18.788487 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:18.788438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:18.960162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:18.960074    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d92c80e-0c04-4553-9e01-08f40da9edf3" path="/var/lib/kubelet/pods/4d92c80e-0c04-4553-9e01-08f40da9edf3/volumes"
Sep 07 11:30:21.832018 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.831977    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"]
Sep 07 11:30:21.832621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.832441    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d92c80e-0c04-4553-9e01-08f40da9edf3" containerName="console"
Sep 07 11:30:21.832621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.832460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d92c80e-0c04-4553-9e01-08f40da9edf3" containerName="console"
Sep 07 11:30:21.832621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.832606    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d92c80e-0c04-4553-9e01-08f40da9edf3" containerName="console"
Sep 07 11:30:21.837127 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.837107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.839511 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.839490    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm44s\""
Sep 07 11:30:21.839610 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.839568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 11:30:21.840258 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.840243    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 11:30:21.845154 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.845131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"]
Sep 07 11:30:21.894056 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.894030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.894166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.894081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwssg\" (UniqueName: \"kubernetes.io/projected/5487f9a0-2704-4c50-9497-f4469aacfc03-kube-api-access-kwssg\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.894166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.894130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.921917 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.921891    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"]
Sep 07 11:30:21.925373 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.925358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:21.933809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.933787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"]
Sep 07 11:30:21.994664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.994639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:21.994792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.994695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.994792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.994726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:21.994928 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.994906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfkrv\" (UniqueName: \"kubernetes.io/projected/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-kube-api-access-zfkrv\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:21.994994 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.994956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwssg\" (UniqueName: \"kubernetes.io/projected/5487f9a0-2704-4c50-9497-f4469aacfc03-kube-api-access-kwssg\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.995059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.995043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.995111 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.995041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:21.995286 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:21.995268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:22.002831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.002811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwssg\" (UniqueName: \"kubernetes.io/projected/5487f9a0-2704-4c50-9497-f4469aacfc03-kube-api-access-kwssg\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:22.026282 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.026259    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"]
Sep 07 11:30:22.029747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.029721    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.038532 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.038509    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"]
Sep 07 11:30:22.095815 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.095753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.095815 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.095795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfkrv\" (UniqueName: \"kubernetes.io/projected/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-kube-api-access-zfkrv\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:22.095965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.095868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.095965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.095919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:22.096085 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.096066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:22.096144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.096134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vpj\" (UniqueName: \"kubernetes.io/projected/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-kube-api-access-64vpj\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.096302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.096287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:22.096372 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.096353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:22.104226 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.104197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfkrv\" (UniqueName: \"kubernetes.io/projected/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-kube-api-access-zfkrv\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:22.121055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.121025    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"]
Sep 07 11:30:22.124606 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.124588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.133075 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.133052    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"]
Sep 07 11:30:22.150142 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.150066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:22.197413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.197374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.197584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.197501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbdb\" (UniqueName: \"kubernetes.io/projected/d1658b03-85ed-46c4-9e65-6ee473d1b11b-kube-api-access-gzbdb\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.197584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.197539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.197690 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.197619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vpj\" (UniqueName: \"kubernetes.io/projected/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-kube-api-access-64vpj\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.197690 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.197674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.197804 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.197758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.198690 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.198200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.199629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.199207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.209499 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.208670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vpj\" (UniqueName: \"kubernetes.io/projected/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-kube-api-access-64vpj\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.234568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.234541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:22.298622 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.298587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbdb\" (UniqueName: \"kubernetes.io/projected/d1658b03-85ed-46c4-9e65-6ee473d1b11b-kube-api-access-gzbdb\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.298736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.298646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.298792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.298769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.299088 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.299065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.299088 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.299082    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.306307 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.306284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbdb\" (UniqueName: \"kubernetes.io/projected/d1658b03-85ed-46c4-9e65-6ee473d1b11b-kube-api-access-gzbdb\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.340225 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.340203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:22.422008 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.421950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"]
Sep 07 11:30:22.424115 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:30:22.424083    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5487f9a0_2704_4c50_9497_f4469aacfc03.slice/crio-fd53c95a5e0c2da8866410b0cc7a4cae27aa8293180d710e0dfe79b60f132392 WatchSource:0}: Error finding container fd53c95a5e0c2da8866410b0cc7a4cae27aa8293180d710e0dfe79b60f132392: Status 404 returned error can't find the container with id fd53c95a5e0c2da8866410b0cc7a4cae27aa8293180d710e0dfe79b60f132392
Sep 07 11:30:22.434091 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.434071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:22.507979 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.507851    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"]
Sep 07 11:30:22.510423 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:30:22.510400    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24ecd2b_3d16_4bbc_a504_821e6c9db2f3.slice/crio-725a19c0a9a6e5e8610b62d5b8794ed5c7794d8eaebbe9285ad20305f681081b WatchSource:0}: Error finding container 725a19c0a9a6e5e8610b62d5b8794ed5c7794d8eaebbe9285ad20305f681081b: Status 404 returned error can't find the container with id 725a19c0a9a6e5e8610b62d5b8794ed5c7794d8eaebbe9285ad20305f681081b
Sep 07 11:30:22.608842 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.608814    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"]
Sep 07 11:30:22.622882 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:30:22.622855    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3eebb5_7298_4c6d_9794_adc3c1471a5f.slice/crio-1ab3eabaaf758dabff411604f8b3d2051dfc9ccab5da3c8d4c9d8541711befd2 WatchSource:0}: Error finding container 1ab3eabaaf758dabff411604f8b3d2051dfc9ccab5da3c8d4c9d8541711befd2: Status 404 returned error can't find the container with id 1ab3eabaaf758dabff411604f8b3d2051dfc9ccab5da3c8d4c9d8541711befd2
Sep 07 11:30:22.705170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:22.705144    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"]
Sep 07 11:30:22.735538 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:30:22.735514    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1658b03_85ed_46c4_9e65_6ee473d1b11b.slice/crio-d296b7cdaa24808551901c650d44b5c3e617803d932cc9679827b83e1f55108c WatchSource:0}: Error finding container d296b7cdaa24808551901c650d44b5c3e617803d932cc9679827b83e1f55108c: Status 404 returned error can't find the container with id d296b7cdaa24808551901c650d44b5c3e617803d932cc9679827b83e1f55108c
Sep 07 11:30:23.040315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.040284    2578 generic.go:356] "Generic (PLEG): container finished" podID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerID="394760fedc616d54fa180f74ca746759321457e208dccbbeea9b1740ed1043a4" exitCode=0
Sep 07 11:30:23.040722 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.040372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w" event={"ID":"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3","Type":"ContainerDied","Data":"394760fedc616d54fa180f74ca746759321457e208dccbbeea9b1740ed1043a4"}
Sep 07 11:30:23.040722 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.040427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w" event={"ID":"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3","Type":"ContainerStarted","Data":"725a19c0a9a6e5e8610b62d5b8794ed5c7794d8eaebbe9285ad20305f681081b"}
Sep 07 11:30:23.042003 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.041981    2578 generic.go:356] "Generic (PLEG): container finished" podID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerID="32dc77253f25d44442883105dd0e40d045814446547fe8cce4cc7d4c5d9b921f" exitCode=0
Sep 07 11:30:23.042079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.042049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp" event={"ID":"d1658b03-85ed-46c4-9e65-6ee473d1b11b","Type":"ContainerDied","Data":"32dc77253f25d44442883105dd0e40d045814446547fe8cce4cc7d4c5d9b921f"}
Sep 07 11:30:23.042143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.042109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp" event={"ID":"d1658b03-85ed-46c4-9e65-6ee473d1b11b","Type":"ContainerStarted","Data":"d296b7cdaa24808551901c650d44b5c3e617803d932cc9679827b83e1f55108c"}
Sep 07 11:30:23.043595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.043576    2578 generic.go:356] "Generic (PLEG): container finished" podID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerID="75779471af0e496960a230023c7186edab7b22ba3c085750bcf9bc7f7bb8d909" exitCode=0
Sep 07 11:30:23.043681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.043659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq" event={"ID":"5487f9a0-2704-4c50-9497-f4469aacfc03","Type":"ContainerDied","Data":"75779471af0e496960a230023c7186edab7b22ba3c085750bcf9bc7f7bb8d909"}
Sep 07 11:30:23.043731 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.043705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq" event={"ID":"5487f9a0-2704-4c50-9497-f4469aacfc03","Type":"ContainerStarted","Data":"fd53c95a5e0c2da8866410b0cc7a4cae27aa8293180d710e0dfe79b60f132392"}
Sep 07 11:30:23.045142 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.045123    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerID="d8baee0cdab3ada56c742d4914fae5db87c3e273f48781668a930735d9b6378c" exitCode=0
Sep 07 11:30:23.045212 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.045148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq" event={"ID":"5b3eebb5-7298-4c6d-9794-adc3c1471a5f","Type":"ContainerDied","Data":"d8baee0cdab3ada56c742d4914fae5db87c3e273f48781668a930735d9b6378c"}
Sep 07 11:30:23.045212 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:23.045179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq" event={"ID":"5b3eebb5-7298-4c6d-9794-adc3c1471a5f","Type":"ContainerStarted","Data":"1ab3eabaaf758dabff411604f8b3d2051dfc9ccab5da3c8d4c9d8541711befd2"}
Sep 07 11:30:24.052439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:24.052409    2578 generic.go:356] "Generic (PLEG): container finished" podID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerID="95e8e3181eceb09be38649886587fb7ec4503c5e86538e03189ebcf33e9320a9" exitCode=0
Sep 07 11:30:24.052750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:24.052485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq" event={"ID":"5487f9a0-2704-4c50-9497-f4469aacfc03","Type":"ContainerDied","Data":"95e8e3181eceb09be38649886587fb7ec4503c5e86538e03189ebcf33e9320a9"}
Sep 07 11:30:25.058678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.058651    2578 generic.go:356] "Generic (PLEG): container finished" podID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerID="a4fc6b491c860b3c0b74a1abd84e9e342aeae450cb5c04551fd5086e09249bb8" exitCode=0
Sep 07 11:30:25.059016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.058742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w" event={"ID":"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3","Type":"ContainerDied","Data":"a4fc6b491c860b3c0b74a1abd84e9e342aeae450cb5c04551fd5086e09249bb8"}
Sep 07 11:30:25.061166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.061130    2578 generic.go:356] "Generic (PLEG): container finished" podID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerID="ce4ff8ca828c26da28d92ad77ddf6c4e00e19c330df01313c80f37c61c37ef97" exitCode=0
Sep 07 11:30:25.061274 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.061233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp" event={"ID":"d1658b03-85ed-46c4-9e65-6ee473d1b11b","Type":"ContainerDied","Data":"ce4ff8ca828c26da28d92ad77ddf6c4e00e19c330df01313c80f37c61c37ef97"}
Sep 07 11:30:25.069941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.069921    2578 generic.go:356] "Generic (PLEG): container finished" podID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerID="ebf97c7f987fd3f1b7e8c9c605d3527468603d405a6627e64e35f79ec3fb0bda" exitCode=0
Sep 07 11:30:25.070039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.070007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq" event={"ID":"5487f9a0-2704-4c50-9497-f4469aacfc03","Type":"ContainerDied","Data":"ebf97c7f987fd3f1b7e8c9c605d3527468603d405a6627e64e35f79ec3fb0bda"}
Sep 07 11:30:25.071848 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.071827    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerID="195fa7374b80357ac7869ffef793124384ffedaca85334749136bcc2e84ab22e" exitCode=0
Sep 07 11:30:25.071934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:25.071853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq" event={"ID":"5b3eebb5-7298-4c6d-9794-adc3c1471a5f","Type":"ContainerDied","Data":"195fa7374b80357ac7869ffef793124384ffedaca85334749136bcc2e84ab22e"}
Sep 07 11:30:26.078706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.078674    2578 generic.go:356] "Generic (PLEG): container finished" podID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerID="27587ccad4ed5085ab6e8aea86e6ff0d8598c3c9442ba22851121fdb89864954" exitCode=0
Sep 07 11:30:26.079058 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.078753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w" event={"ID":"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3","Type":"ContainerDied","Data":"27587ccad4ed5085ab6e8aea86e6ff0d8598c3c9442ba22851121fdb89864954"}
Sep 07 11:30:26.080821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.080798    2578 generic.go:356] "Generic (PLEG): container finished" podID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerID="7f1b4acc2a7e4defbf4896f839ba7d50e75fb4498ca2fb02e1ab43cb3a5e6240" exitCode=0
Sep 07 11:30:26.080935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.080872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp" event={"ID":"d1658b03-85ed-46c4-9e65-6ee473d1b11b","Type":"ContainerDied","Data":"7f1b4acc2a7e4defbf4896f839ba7d50e75fb4498ca2fb02e1ab43cb3a5e6240"}
Sep 07 11:30:26.083035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.083015    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerID="cfd9c799dedc607d8159a36c02efa68b195db58443121a903270d6de7fb6177d" exitCode=0
Sep 07 11:30:26.083146 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.083077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq" event={"ID":"5b3eebb5-7298-4c6d-9794-adc3c1471a5f","Type":"ContainerDied","Data":"cfd9c799dedc607d8159a36c02efa68b195db58443121a903270d6de7fb6177d"}
Sep 07 11:30:26.209930 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.209880    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:26.337849 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.337820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwssg\" (UniqueName: \"kubernetes.io/projected/5487f9a0-2704-4c50-9497-f4469aacfc03-kube-api-access-kwssg\") pod \"5487f9a0-2704-4c50-9497-f4469aacfc03\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") "
Sep 07 11:30:26.337988 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.337897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-bundle\") pod \"5487f9a0-2704-4c50-9497-f4469aacfc03\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") "
Sep 07 11:30:26.337988 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.337947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-util\") pod \"5487f9a0-2704-4c50-9497-f4469aacfc03\" (UID: \"5487f9a0-2704-4c50-9497-f4469aacfc03\") "
Sep 07 11:30:26.338412 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.338372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-bundle" (OuterVolumeSpecName: "bundle") pod "5487f9a0-2704-4c50-9497-f4469aacfc03" (UID: "5487f9a0-2704-4c50-9497-f4469aacfc03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:26.339800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.339778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5487f9a0-2704-4c50-9497-f4469aacfc03-kube-api-access-kwssg" (OuterVolumeSpecName: "kube-api-access-kwssg") pod "5487f9a0-2704-4c50-9497-f4469aacfc03" (UID: "5487f9a0-2704-4c50-9497-f4469aacfc03"). InnerVolumeSpecName "kube-api-access-kwssg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:30:26.343602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.343576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-util" (OuterVolumeSpecName: "util") pod "5487f9a0-2704-4c50-9497-f4469aacfc03" (UID: "5487f9a0-2704-4c50-9497-f4469aacfc03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:26.438502 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.438460    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:26.438502 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.438503    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5487f9a0-2704-4c50-9497-f4469aacfc03-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:26.438651 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:26.438513    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwssg\" (UniqueName: \"kubernetes.io/projected/5487f9a0-2704-4c50-9497-f4469aacfc03-kube-api-access-kwssg\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.090079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.090049    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq"
Sep 07 11:30:27.090079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.090068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbb4ggq" event={"ID":"5487f9a0-2704-4c50-9497-f4469aacfc03","Type":"ContainerDied","Data":"fd53c95a5e0c2da8866410b0cc7a4cae27aa8293180d710e0dfe79b60f132392"}
Sep 07 11:30:27.090628 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.090125    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd53c95a5e0c2da8866410b0cc7a4cae27aa8293180d710e0dfe79b60f132392"
Sep 07 11:30:27.250961 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.250894    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:27.253906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.253868    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:27.278398 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.278377    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:27.346168 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-util\") pod \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") "
Sep 07 11:30:27.346295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-bundle\") pod \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") "
Sep 07 11:30:27.346295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfkrv\" (UniqueName: \"kubernetes.io/projected/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-kube-api-access-zfkrv\") pod \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") "
Sep 07 11:30:27.346295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-util\") pod \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") "
Sep 07 11:30:27.346295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64vpj\" (UniqueName: \"kubernetes.io/projected/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-kube-api-access-64vpj\") pod \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") "
Sep 07 11:30:27.346295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzbdb\" (UniqueName: \"kubernetes.io/projected/d1658b03-85ed-46c4-9e65-6ee473d1b11b-kube-api-access-gzbdb\") pod \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\" (UID: \"d1658b03-85ed-46c4-9e65-6ee473d1b11b\") "
Sep 07 11:30:27.346600 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346304    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-util\") pod \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") "
Sep 07 11:30:27.346600 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-bundle\") pod \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\" (UID: \"5b3eebb5-7298-4c6d-9794-adc3c1471a5f\") "
Sep 07 11:30:27.346600 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.346408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-bundle\") pod \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\" (UID: \"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3\") "
Sep 07 11:30:27.347408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.347089    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-bundle" (OuterVolumeSpecName: "bundle") pod "d1658b03-85ed-46c4-9e65-6ee473d1b11b" (UID: "d1658b03-85ed-46c4-9e65-6ee473d1b11b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:27.347408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.347158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-bundle" (OuterVolumeSpecName: "bundle") pod "c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" (UID: "c24ecd2b-3d16-4bbc-a504-821e6c9db2f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:27.347655 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.347618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-bundle" (OuterVolumeSpecName: "bundle") pod "5b3eebb5-7298-4c6d-9794-adc3c1471a5f" (UID: "5b3eebb5-7298-4c6d-9794-adc3c1471a5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:27.349002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.348973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1658b03-85ed-46c4-9e65-6ee473d1b11b-kube-api-access-gzbdb" (OuterVolumeSpecName: "kube-api-access-gzbdb") pod "d1658b03-85ed-46c4-9e65-6ee473d1b11b" (UID: "d1658b03-85ed-46c4-9e65-6ee473d1b11b"). InnerVolumeSpecName "kube-api-access-gzbdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:30:27.349091 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.349036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-kube-api-access-64vpj" (OuterVolumeSpecName: "kube-api-access-64vpj") pod "5b3eebb5-7298-4c6d-9794-adc3c1471a5f" (UID: "5b3eebb5-7298-4c6d-9794-adc3c1471a5f"). InnerVolumeSpecName "kube-api-access-64vpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:30:27.349091 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.349063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-kube-api-access-zfkrv" (OuterVolumeSpecName: "kube-api-access-zfkrv") pod "c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" (UID: "c24ecd2b-3d16-4bbc-a504-821e6c9db2f3"). InnerVolumeSpecName "kube-api-access-zfkrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:30:27.352634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.352611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-util" (OuterVolumeSpecName: "util") pod "c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" (UID: "c24ecd2b-3d16-4bbc-a504-821e6c9db2f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:27.352737 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.352688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-util" (OuterVolumeSpecName: "util") pod "d1658b03-85ed-46c4-9e65-6ee473d1b11b" (UID: "d1658b03-85ed-46c4-9e65-6ee473d1b11b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:27.353054 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.353030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-util" (OuterVolumeSpecName: "util") pod "5b3eebb5-7298-4c6d-9794-adc3c1471a5f" (UID: "5b3eebb5-7298-4c6d-9794-adc3c1471a5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:30:27.447969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.447945    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.447969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.447967    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.448092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.447977    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1658b03-85ed-46c4-9e65-6ee473d1b11b-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.448092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.447986    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfkrv\" (UniqueName: \"kubernetes.io/projected/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-kube-api-access-zfkrv\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.448092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.447994    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24ecd2b-3d16-4bbc-a504-821e6c9db2f3-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.448092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.448003    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vpj\" (UniqueName: \"kubernetes.io/projected/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-kube-api-access-64vpj\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.448092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.448011    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbdb\" (UniqueName: \"kubernetes.io/projected/d1658b03-85ed-46c4-9e65-6ee473d1b11b-kube-api-access-gzbdb\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.448092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.448019    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-util\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:27.448092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:27.448027    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b3eebb5-7298-4c6d-9794-adc3c1471a5f-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:30:28.103064 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.103030    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp"
Sep 07 11:30:28.103064 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.103051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wp6fp" event={"ID":"d1658b03-85ed-46c4-9e65-6ee473d1b11b","Type":"ContainerDied","Data":"d296b7cdaa24808551901c650d44b5c3e617803d932cc9679827b83e1f55108c"}
Sep 07 11:30:28.103568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.103132    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d296b7cdaa24808551901c650d44b5c3e617803d932cc9679827b83e1f55108c"
Sep 07 11:30:28.104919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.104895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq"
Sep 07 11:30:28.105032 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.104914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a072czq" event={"ID":"5b3eebb5-7298-4c6d-9794-adc3c1471a5f","Type":"ContainerDied","Data":"1ab3eabaaf758dabff411604f8b3d2051dfc9ccab5da3c8d4c9d8541711befd2"}
Sep 07 11:30:28.105032 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.104957    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ab3eabaaf758dabff411604f8b3d2051dfc9ccab5da3c8d4c9d8541711befd2"
Sep 07 11:30:28.106847 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.106827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w" event={"ID":"c24ecd2b-3d16-4bbc-a504-821e6c9db2f3","Type":"ContainerDied","Data":"725a19c0a9a6e5e8610b62d5b8794ed5c7794d8eaebbe9285ad20305f681081b"}
Sep 07 11:30:28.106934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.106858    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="725a19c0a9a6e5e8610b62d5b8794ed5c7794d8eaebbe9285ad20305f681081b"
Sep 07 11:30:28.106934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:28.106858    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cwvf7w"
Sep 07 11:30:42.098346 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:42.098317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:42.902071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:42.902032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:43.756079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:43.756044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:46.767250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:46.767221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:46.867173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:46.867139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:46.973513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:46.973481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:47.068069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:47.067995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:47.450204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:47.450168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:48.354209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:48.354178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:49.936752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:49.936724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:50.025176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:50.025147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:52.141417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:52.141384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:53.658481 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:53.657868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:54.515695 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.515662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:54.528111 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528085    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"]
Sep 07 11:30:54.528400 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528388    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerName="pull"
Sep 07 11:30:54.528442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528402    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerName="pull"
Sep 07 11:30:54.528442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528410    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerName="extract"
Sep 07 11:30:54.528442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528416    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerName="extract"
Sep 07 11:30:54.528442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerName="util"
Sep 07 11:30:54.528442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528430    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerName="util"
Sep 07 11:30:54.528442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528437    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerName="util"
Sep 07 11:30:54.528442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528443    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerName="util"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528449    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerName="pull"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528455    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerName="pull"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528481    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528489    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528501    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerName="pull"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerName="pull"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528517    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerName="pull"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528524    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerName="pull"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528538    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528544    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528551    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerName="util"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528556    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerName="util"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528561    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528567    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528574    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerName="util"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528578    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerName="util"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528643    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1658b03-85ed-46c4-9e65-6ee473d1b11b" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528651    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24ecd2b-3d16-4bbc-a504-821e6c9db2f3" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528660    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b3eebb5-7298-4c6d-9794-adc3c1471a5f" containerName="extract"
Sep 07 11:30:54.528680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.528668    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5487f9a0-2704-4c50-9497-f4469aacfc03" containerName="extract"
Sep 07 11:30:54.532824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.532807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.539413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.539393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 07 11:30:54.539537 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.539434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xk74w\""
Sep 07 11:30:54.539925 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.539909    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 11:30:54.540565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.540349    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 07 11:30:54.540565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.540418    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 11:30:54.546790 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.546770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"]
Sep 07 11:30:54.665924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.665896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whcv\" (UniqueName: \"kubernetes.io/projected/0d8261f2-86a8-44d8-afbf-575adedbf159-kube-api-access-7whcv\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.666261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.665936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d8261f2-86a8-44d8-afbf-575adedbf159-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.666261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.665973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8261f2-86a8-44d8-afbf-575adedbf159-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.767043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.766971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whcv\" (UniqueName: \"kubernetes.io/projected/0d8261f2-86a8-44d8-afbf-575adedbf159-kube-api-access-7whcv\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.767043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.767006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d8261f2-86a8-44d8-afbf-575adedbf159-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.767043 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.767040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8261f2-86a8-44d8-afbf-575adedbf159-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.767762 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.767739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d8261f2-86a8-44d8-afbf-575adedbf159-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.769411 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.769384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8261f2-86a8-44d8-afbf-575adedbf159-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.775433 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.775410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whcv\" (UniqueName: \"kubernetes.io/projected/0d8261f2-86a8-44d8-afbf-575adedbf159-kube-api-access-7whcv\") pod \"kuadrant-console-plugin-598f86468b-7rk94\" (UID: \"0d8261f2-86a8-44d8-afbf-575adedbf159\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:54.790130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.790106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:54.843346 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:54.843325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"
Sep 07 11:30:55.143957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:55.143927    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94"]
Sep 07 11:30:55.145482 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:30:55.145436    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d8261f2_86a8_44d8_afbf_575adedbf159.slice/crio-662e73552105cd2501779ab34860b8342dbf442b940a991379303c41d03267bd WatchSource:0}: Error finding container 662e73552105cd2501779ab34860b8342dbf442b940a991379303c41d03267bd: Status 404 returned error can't find the container with id 662e73552105cd2501779ab34860b8342dbf442b940a991379303c41d03267bd
Sep 07 11:30:55.241810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:55.241780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94" event={"ID":"0d8261f2-86a8-44d8-afbf-575adedbf159","Type":"ContainerStarted","Data":"662e73552105cd2501779ab34860b8342dbf442b940a991379303c41d03267bd"}
Sep 07 11:30:57.318070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:57.318025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:57.403962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:57.403925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:57.714994 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:57.714965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:57.817987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:57.817957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:58.110060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:58.109974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:30:58.174065 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:30:58.174027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:00.274178 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:00.274138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94" event={"ID":"0d8261f2-86a8-44d8-afbf-575adedbf159","Type":"ContainerStarted","Data":"17974a0ab354f988b9ff5094fd185b37e7a1bca7c284629fea738c1570356672"}
Sep 07 11:31:00.291708 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:00.291655    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-7rk94" podStartSLOduration=1.6953352929999999 podStartE2EDuration="6.2916419s" totalImagesPullingTime="4.596306607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:31:00.290171046 +0000 UTC m=+491.765019194" watchObservedRunningTime="2026-09-07 11:31:00.2916419 +0000 UTC m=+491.766490052"
Sep 07 11:31:05.867867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:05.867834    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"]
Sep 07 11:31:05.872321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:05.872297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:05.874839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:05.874818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6brmr\""
Sep 07 11:31:05.889365 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:05.889343    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"]
Sep 07 11:31:05.959780 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:05.959748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-t5mms\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:05.959943 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:05.959807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgjm\" (UniqueName: \"kubernetes.io/projected/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-kube-api-access-lrgjm\") pod \"kuadrant-operator-controller-manager-68cf589589-t5mms\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:06.060720 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.060677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-t5mms\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:06.060885 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.060739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgjm\" (UniqueName: \"kubernetes.io/projected/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-kube-api-access-lrgjm\") pod \"kuadrant-operator-controller-manager-68cf589589-t5mms\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:06.061129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.061110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-t5mms\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:06.095391 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.095356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgjm\" (UniqueName: \"kubernetes.io/projected/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-kube-api-access-lrgjm\") pod \"kuadrant-operator-controller-manager-68cf589589-t5mms\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:06.184652 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.184561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:06.216505 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.216433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:06.323791 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.323752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:06.463264 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:06.463234    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"]
Sep 07 11:31:06.465445 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:31:06.465415    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffebeeb1_bc7b_42dc_93a3_22f9e093333a.slice/crio-fc091beb9251a3b6e5b32772036ca7b7137bff900aa496e05d2fc9ea57110115 WatchSource:0}: Error finding container fc091beb9251a3b6e5b32772036ca7b7137bff900aa496e05d2fc9ea57110115: Status 404 returned error can't find the container with id fc091beb9251a3b6e5b32772036ca7b7137bff900aa496e05d2fc9ea57110115
Sep 07 11:31:07.314736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:07.314695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms" event={"ID":"ffebeeb1-bc7b-42dc-93a3-22f9e093333a","Type":"ContainerStarted","Data":"fc091beb9251a3b6e5b32772036ca7b7137bff900aa496e05d2fc9ea57110115"}
Sep 07 11:31:08.584301 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:08.584263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:08.685473 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:08.685425    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:10.334069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:10.334032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms" event={"ID":"ffebeeb1-bc7b-42dc-93a3-22f9e093333a","Type":"ContainerStarted","Data":"ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04"}
Sep 07 11:31:10.334412 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:10.334147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:10.365435 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:10.365384    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms" podStartSLOduration=1.6029866560000001 podStartE2EDuration="5.365367847s" totalImagesPullingTime="3.762381191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:31:10.362882649 +0000 UTC m=+501.837730797" watchObservedRunningTime="2026-09-07 11:31:10.365367847 +0000 UTC m=+501.840215995"
Sep 07 11:31:13.318789 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:13.318752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:13.420483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:13.420429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:18.537438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:18.537398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:18.609014 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:18.608958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:18.954984 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:18.954938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:19.063505 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:19.063448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:19.156156 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:19.156118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:19.259999 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:19.259894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:21.341597 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:21.341569    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:25.848187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:25.848148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:25.948385 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:25.948340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:27.935938 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:27.935898    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:28.034945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:28.034906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:30.388358 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:30.388316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:30.493062 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:30.493026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:35.135327 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:35.130205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:35.230738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:35.230703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:37.062781 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.062742    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"]
Sep 07 11:31:37.063209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.063039    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms" podUID="ffebeeb1-bc7b-42dc-93a3-22f9e093333a" containerName="manager" containerID="cri-o://ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04" gracePeriod=10
Sep 07 11:31:37.313977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.313916    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:37.431543 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.431501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrgjm\" (UniqueName: \"kubernetes.io/projected/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-kube-api-access-lrgjm\") pod \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") "
Sep 07 11:31:37.431725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.431618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-extensions-socket-volume\") pod \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\" (UID: \"ffebeeb1-bc7b-42dc-93a3-22f9e093333a\") "
Sep 07 11:31:37.432113 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.432082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ffebeeb1-bc7b-42dc-93a3-22f9e093333a" (UID: "ffebeeb1-bc7b-42dc-93a3-22f9e093333a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:31:37.433727 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.433691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-kube-api-access-lrgjm" (OuterVolumeSpecName: "kube-api-access-lrgjm") pod "ffebeeb1-bc7b-42dc-93a3-22f9e093333a" (UID: "ffebeeb1-bc7b-42dc-93a3-22f9e093333a"). InnerVolumeSpecName "kube-api-access-lrgjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:31:37.483316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.483279    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffebeeb1-bc7b-42dc-93a3-22f9e093333a" containerID="ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04" exitCode=0
Sep 07 11:31:37.483494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.483319    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"
Sep 07 11:31:37.483494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.483319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms" event={"ID":"ffebeeb1-bc7b-42dc-93a3-22f9e093333a","Type":"ContainerDied","Data":"ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04"}
Sep 07 11:31:37.483494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.483452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms" event={"ID":"ffebeeb1-bc7b-42dc-93a3-22f9e093333a","Type":"ContainerDied","Data":"fc091beb9251a3b6e5b32772036ca7b7137bff900aa496e05d2fc9ea57110115"}
Sep 07 11:31:37.483607 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.483516    2578 scope.go:117] "RemoveContainer" containerID="ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04"
Sep 07 11:31:37.492942 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.492923    2578 scope.go:117] "RemoveContainer" containerID="ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04"
Sep 07 11:31:37.493196 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:31:37.493178    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04\": container with ID starting with ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04 not found: ID does not exist" containerID="ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04"
Sep 07 11:31:37.493241 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.493203    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04"} err="failed to get container status \"ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04\": rpc error: code = NotFound desc = could not find container \"ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04\": container with ID starting with ed37038ac1a15be4303189363f11da02d8de293211b34f088109869c9ba3ea04 not found: ID does not exist"
Sep 07 11:31:37.512007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.511982    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"]
Sep 07 11:31:37.518163 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.518135    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-t5mms"]
Sep 07 11:31:37.532740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.532719    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgjm\" (UniqueName: \"kubernetes.io/projected/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-kube-api-access-lrgjm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:31:37.532828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.532742    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffebeeb1-bc7b-42dc-93a3-22f9e093333a-extensions-socket-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:31:37.826904 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.826868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:37.951068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:37.951033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:38.960262 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:38.960227    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffebeeb1-bc7b-42dc-93a3-22f9e093333a" path="/var/lib/kubelet/pods/ffebeeb1-bc7b-42dc-93a3-22f9e093333a/volumes"
Sep 07 11:31:54.205761 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.205717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:54.307852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.307822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:54.410674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.410645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:54.571821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.571740    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:31:54.572113 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.572098    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffebeeb1-bc7b-42dc-93a3-22f9e093333a" containerName="manager"
Sep 07 11:31:54.572251 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.572114    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffebeeb1-bc7b-42dc-93a3-22f9e093333a" containerName="manager"
Sep 07 11:31:54.572251 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.572178    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffebeeb1-bc7b-42dc-93a3-22f9e093333a" containerName="manager"
Sep 07 11:31:54.576171 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.576151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:54.589931 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.589910    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 11:31:54.591836 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.591816    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:31:54.626165 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.626138    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:31:54.686123 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.686094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ps2t\" (UniqueName: \"kubernetes.io/projected/4b145e05-ef9e-49ec-b44e-04d283e34fd3-kube-api-access-7ps2t\") pod \"limitador-limitador-85d9db4d5d-kx7sm\" (UID: \"4b145e05-ef9e-49ec-b44e-04d283e34fd3\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:54.686265 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.686141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b145e05-ef9e-49ec-b44e-04d283e34fd3-config-file\") pod \"limitador-limitador-85d9db4d5d-kx7sm\" (UID: \"4b145e05-ef9e-49ec-b44e-04d283e34fd3\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:54.786584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.786549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ps2t\" (UniqueName: \"kubernetes.io/projected/4b145e05-ef9e-49ec-b44e-04d283e34fd3-kube-api-access-7ps2t\") pod \"limitador-limitador-85d9db4d5d-kx7sm\" (UID: \"4b145e05-ef9e-49ec-b44e-04d283e34fd3\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:54.786765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.786595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b145e05-ef9e-49ec-b44e-04d283e34fd3-config-file\") pod \"limitador-limitador-85d9db4d5d-kx7sm\" (UID: \"4b145e05-ef9e-49ec-b44e-04d283e34fd3\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:54.787228 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.787201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b145e05-ef9e-49ec-b44e-04d283e34fd3-config-file\") pod \"limitador-limitador-85d9db4d5d-kx7sm\" (UID: \"4b145e05-ef9e-49ec-b44e-04d283e34fd3\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:54.796052 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.796026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ps2t\" (UniqueName: \"kubernetes.io/projected/4b145e05-ef9e-49ec-b44e-04d283e34fd3-kube-api-access-7ps2t\") pod \"limitador-limitador-85d9db4d5d-kx7sm\" (UID: \"4b145e05-ef9e-49ec-b44e-04d283e34fd3\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:54.885987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:54.885958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:55.070180 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.070145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8kvc"]
Sep 07 11:31:55.073705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.073682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8kvc"
Sep 07 11:31:55.079028 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.079002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wr2st\""
Sep 07 11:31:55.090666 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.090646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8kvc"]
Sep 07 11:31:55.130161 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.130133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:55.166930 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.166904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:31:55.168172 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:31:55.168148    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b145e05_ef9e_49ec_b44e_04d283e34fd3.slice/crio-cb534704dfe6ce6267e6602233606f0072c6108607d6f64044f1b8eea5a0f3e0 WatchSource:0}: Error finding container cb534704dfe6ce6267e6602233606f0072c6108607d6f64044f1b8eea5a0f3e0: Status 404 returned error can't find the container with id cb534704dfe6ce6267e6602233606f0072c6108607d6f64044f1b8eea5a0f3e0
Sep 07 11:31:55.189605 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.189577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9bq\" (UniqueName: \"kubernetes.io/projected/ba86a0ff-2815-4393-9dfe-e41acdeafed0-kube-api-access-4q9bq\") pod \"authorino-b54df94b4-b8kvc\" (UID: \"ba86a0ff-2815-4393-9dfe-e41acdeafed0\") " pod="kuadrant-system/authorino-b54df94b4-b8kvc"
Sep 07 11:31:55.237557 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.237530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:31:55.290853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.290825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9bq\" (UniqueName: \"kubernetes.io/projected/ba86a0ff-2815-4393-9dfe-e41acdeafed0-kube-api-access-4q9bq\") pod \"authorino-b54df94b4-b8kvc\" (UID: \"ba86a0ff-2815-4393-9dfe-e41acdeafed0\") " pod="kuadrant-system/authorino-b54df94b4-b8kvc"
Sep 07 11:31:55.301839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.301808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9bq\" (UniqueName: \"kubernetes.io/projected/ba86a0ff-2815-4393-9dfe-e41acdeafed0-kube-api-access-4q9bq\") pod \"authorino-b54df94b4-b8kvc\" (UID: \"ba86a0ff-2815-4393-9dfe-e41acdeafed0\") " pod="kuadrant-system/authorino-b54df94b4-b8kvc"
Sep 07 11:31:55.382557 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.382531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8kvc"
Sep 07 11:31:55.580969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.580933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm" event={"ID":"4b145e05-ef9e-49ec-b44e-04d283e34fd3","Type":"ContainerStarted","Data":"cb534704dfe6ce6267e6602233606f0072c6108607d6f64044f1b8eea5a0f3e0"}
Sep 07 11:31:55.660743 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:55.660709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8kvc"]
Sep 07 11:31:55.663034 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:31:55.663008    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba86a0ff_2815_4393_9dfe_e41acdeafed0.slice/crio-72687d616b02f719945299c025ca61087edb61e41a61a8f62e88a8ac69294789 WatchSource:0}: Error finding container 72687d616b02f719945299c025ca61087edb61e41a61a8f62e88a8ac69294789: Status 404 returned error can't find the container with id 72687d616b02f719945299c025ca61087edb61e41a61a8f62e88a8ac69294789
Sep 07 11:31:56.592911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:56.592821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm" event={"ID":"4b145e05-ef9e-49ec-b44e-04d283e34fd3","Type":"ContainerStarted","Data":"6e7544914f64fadf760e1cf9618660a049206549732fa56367ab222f0e48bf46"}
Sep 07 11:31:56.593354 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:56.592918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:31:56.594525 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:56.594495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8kvc" event={"ID":"ba86a0ff-2815-4393-9dfe-e41acdeafed0","Type":"ContainerStarted","Data":"72687d616b02f719945299c025ca61087edb61e41a61a8f62e88a8ac69294789"}
Sep 07 11:31:56.614155 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:56.614095    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm" podStartSLOduration=1.573280895 podStartE2EDuration="2.614076145s" totalImagesPullingTime="1.04079525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:31:56.613158363 +0000 UTC m=+548.088006533" watchObservedRunningTime="2026-09-07 11:31:56.614076145 +0000 UTC m=+548.088924296"
Sep 07 11:31:57.837963 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:57.837932    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8kvc"]
Sep 07 11:31:58.606022 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:58.605979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8kvc" event={"ID":"ba86a0ff-2815-4393-9dfe-e41acdeafed0","Type":"ContainerStarted","Data":"e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a"}
Sep 07 11:31:58.628508 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:58.628444    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-b8kvc" podStartSLOduration=1.68938328 podStartE2EDuration="3.628430481s" totalImagesPullingTime="1.939047201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:31:58.625005108 +0000 UTC m=+550.099853257" watchObservedRunningTime="2026-09-07 11:31:58.628430481 +0000 UTC m=+550.103278651"
Sep 07 11:31:59.610686 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:59.610642    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-b8kvc" podUID="ba86a0ff-2815-4393-9dfe-e41acdeafed0" containerName="authorino" containerID="cri-o://e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a" gracePeriod=30
Sep 07 11:31:59.859597 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:59.859573    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8kvc"
Sep 07 11:31:59.931705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:59.931639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q9bq\" (UniqueName: \"kubernetes.io/projected/ba86a0ff-2815-4393-9dfe-e41acdeafed0-kube-api-access-4q9bq\") pod \"ba86a0ff-2815-4393-9dfe-e41acdeafed0\" (UID: \"ba86a0ff-2815-4393-9dfe-e41acdeafed0\") "
Sep 07 11:31:59.933630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:31:59.933604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba86a0ff-2815-4393-9dfe-e41acdeafed0-kube-api-access-4q9bq" (OuterVolumeSpecName: "kube-api-access-4q9bq") pod "ba86a0ff-2815-4393-9dfe-e41acdeafed0" (UID: "ba86a0ff-2815-4393-9dfe-e41acdeafed0"). InnerVolumeSpecName "kube-api-access-4q9bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:32:00.032990 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.032951    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q9bq\" (UniqueName: \"kubernetes.io/projected/ba86a0ff-2815-4393-9dfe-e41acdeafed0-kube-api-access-4q9bq\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:32:00.617213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.617179    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba86a0ff-2815-4393-9dfe-e41acdeafed0" containerID="e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a" exitCode=0
Sep 07 11:32:00.617634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.617217    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8kvc"
Sep 07 11:32:00.617634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.617264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8kvc" event={"ID":"ba86a0ff-2815-4393-9dfe-e41acdeafed0","Type":"ContainerDied","Data":"e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a"}
Sep 07 11:32:00.617634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.617307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8kvc" event={"ID":"ba86a0ff-2815-4393-9dfe-e41acdeafed0","Type":"ContainerDied","Data":"72687d616b02f719945299c025ca61087edb61e41a61a8f62e88a8ac69294789"}
Sep 07 11:32:00.617634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.617327    2578 scope.go:117] "RemoveContainer" containerID="e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a"
Sep 07 11:32:00.626250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.626233    2578 scope.go:117] "RemoveContainer" containerID="e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a"
Sep 07 11:32:00.626524 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:32:00.626505    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a\": container with ID starting with e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a not found: ID does not exist" containerID="e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a"
Sep 07 11:32:00.626589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.626532    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a"} err="failed to get container status \"e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a\": rpc error: code = NotFound desc = could not find container \"e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a\": container with ID starting with e3c6672155fa7a52c0034ae3246335b4669f8c43c07b9a337c0466281657df0a not found: ID does not exist"
Sep 07 11:32:00.638721 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.638695    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8kvc"]
Sep 07 11:32:00.642518 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.642496    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8kvc"]
Sep 07 11:32:00.959514 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:00.959424    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba86a0ff-2815-4393-9dfe-e41acdeafed0" path="/var/lib/kubelet/pods/ba86a0ff-2815-4393-9dfe-e41acdeafed0/volumes"
Sep 07 11:32:07.601220 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:07.601189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"
Sep 07 11:32:11.544633 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:11.544597    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:11.644112 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:11.644078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:12.008240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:12.008208    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:24.281014 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:24.280980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:24.651328 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:24.651291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:32.137989 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:32.137940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:32.221940 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:32.221903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:32.316955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:32.316915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:32.402036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:32.401954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:35.372645 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:35.372607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:35.502667 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:35.502634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:38.115943 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:38.115902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:32:49.301802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:49.301637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:32:49.307934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:49.302656    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:32:49.307934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:49.306340    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:32:49.307934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:32:49.306846    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:33:07.244514 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:07.244480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:07.345630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:07.345597    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:08.710746 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:08.710712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:08.824000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:08.823963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:08.926744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:08.926708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:09.019699 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:09.019615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:09.380811 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:09.380780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:09.479452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:09.479415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:09.722139 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:09.722061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:33:18.621763 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:18.621732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:18.696864 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:18.696830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:25.596332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.596291    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"]
Sep 07 11:33:25.596828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.596757    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba86a0ff-2815-4393-9dfe-e41acdeafed0" containerName="authorino"
Sep 07 11:33:25.596828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.596775    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba86a0ff-2815-4393-9dfe-e41acdeafed0" containerName="authorino"
Sep 07 11:33:25.596901 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.596833    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba86a0ff-2815-4393-9dfe-e41acdeafed0" containerName="authorino"
Sep 07 11:33:25.599786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.599769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.602307 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.602283    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 11:33:25.603118 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.603095    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bf2b6\""
Sep 07 11:33:25.603226 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.603164    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 07 11:33:25.603293 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.603257    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 11:33:25.609905 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.609884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"]
Sep 07 11:33:25.625086 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.625063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.625188 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.625102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.625188 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.625124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.625188 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.625180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsdn\" (UniqueName: \"kubernetes.io/projected/d6c66737-a29b-46fb-9d79-6de216f7f94f-kube-api-access-4wsdn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.625310 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.625206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.625310 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.625229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c66737-a29b-46fb-9d79-6de216f7f94f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.700439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.700407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:25.726229 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsdn\" (UniqueName: \"kubernetes.io/projected/d6c66737-a29b-46fb-9d79-6de216f7f94f-kube-api-access-4wsdn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c66737-a29b-46fb-9d79-6de216f7f94f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.726856 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.726768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.728521 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.728495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c66737-a29b-46fb-9d79-6de216f7f94f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.728782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.728763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c66737-a29b-46fb-9d79-6de216f7f94f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.735136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.735117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsdn\" (UniqueName: \"kubernetes.io/projected/d6c66737-a29b-46fb-9d79-6de216f7f94f-kube-api-access-4wsdn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl\" (UID: \"d6c66737-a29b-46fb-9d79-6de216f7f94f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:25.911023 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:25.910988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:26.102184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:26.102153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:33:26.182966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:26.182936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"]
Sep 07 11:33:26.184842 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:33:26.184814    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c66737_a29b_46fb_9d79_6de216f7f94f.slice/crio-ded027e68dd6f4c16c6b8d504d914688fd88e0fe0ac2efe29e69edea03ae4713 WatchSource:0}: Error finding container ded027e68dd6f4c16c6b8d504d914688fd88e0fe0ac2efe29e69edea03ae4713: Status 404 returned error can't find the container with id ded027e68dd6f4c16c6b8d504d914688fd88e0fe0ac2efe29e69edea03ae4713
Sep 07 11:33:26.186088 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:26.186067    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:33:27.059312 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:27.059266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl" event={"ID":"d6c66737-a29b-46fb-9d79-6de216f7f94f","Type":"ContainerStarted","Data":"ded027e68dd6f4c16c6b8d504d914688fd88e0fe0ac2efe29e69edea03ae4713"}
Sep 07 11:33:28.066136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:28.066091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl" event={"ID":"d6c66737-a29b-46fb-9d79-6de216f7f94f","Type":"ContainerStarted","Data":"6705f9c31e121dc9e0d32332254e8dda3fd6fccf366bc13f3100e0782cb146eb"}
Sep 07 11:33:28.085848 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:28.085801    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl" podStartSLOduration=2.047470157 podStartE2EDuration="3.085778211s" totalImagesPullingTime="1.038308054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:33:28.085279749 +0000 UTC m=+639.560127926" watchObservedRunningTime="2026-09-07 11:33:28.085778211 +0000 UTC m=+639.560626361"
Sep 07 11:33:30.325608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:30.325574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:31.011929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:31.011893    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:33:35.911641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:35.911606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:35.912018 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:35.911666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:35.924289 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:35.924260    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:36.121273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:36.121242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl"
Sep 07 11:33:39.206953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:39.206903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:33:39.698917 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:33:39.698868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:34:03.088364 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:03.088334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:34:03.210159 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:03.210125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:34:04.306971 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:04.306937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:34:07.163589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:07.163548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:34:22.245664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:22.245630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:34:23.704813 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:23.704779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:34:31.525347 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:31.525313    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:34:33.200422 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:33.200383    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:34:38.645073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:38.645039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:34:41.594904 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:34:41.594865    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:35:32.665314 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:32.665280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:33.254775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:33.254741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:41.055716 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.055677    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6677b4689b-sx8j5"]
Sep 07 11:35:41.059332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.059307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.062595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.062572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wr2st\""
Sep 07 11:35:41.062679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.062597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 07 11:35:41.072475 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.072441    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6677b4689b-sx8j5"]
Sep 07 11:35:41.159633 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.159594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g994c\" (UniqueName: \"kubernetes.io/projected/9eac05d8-8987-4b06-ada2-9d3c9de2c0b6-kube-api-access-g994c\") pod \"authorino-6677b4689b-sx8j5\" (UID: \"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6\") " pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.159803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.159668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9eac05d8-8987-4b06-ada2-9d3c9de2c0b6-tls-cert\") pod \"authorino-6677b4689b-sx8j5\" (UID: \"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6\") " pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.260350 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.260314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g994c\" (UniqueName: \"kubernetes.io/projected/9eac05d8-8987-4b06-ada2-9d3c9de2c0b6-kube-api-access-g994c\") pod \"authorino-6677b4689b-sx8j5\" (UID: \"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6\") " pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.260544 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.260389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9eac05d8-8987-4b06-ada2-9d3c9de2c0b6-tls-cert\") pod \"authorino-6677b4689b-sx8j5\" (UID: \"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6\") " pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.262757 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.262732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9eac05d8-8987-4b06-ada2-9d3c9de2c0b6-tls-cert\") pod \"authorino-6677b4689b-sx8j5\" (UID: \"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6\") " pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.268029 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.268005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g994c\" (UniqueName: \"kubernetes.io/projected/9eac05d8-8987-4b06-ada2-9d3c9de2c0b6-kube-api-access-g994c\") pod \"authorino-6677b4689b-sx8j5\" (UID: \"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6\") " pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.368046 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.368015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6677b4689b-sx8j5"
Sep 07 11:35:41.637071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.637046    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6677b4689b-sx8j5"]
Sep 07 11:35:41.639192 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:35:41.639163    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eac05d8_8987_4b06_ada2_9d3c9de2c0b6.slice/crio-e999b181aaec7248e4ee3ac9fd391d61a4a0ef65d1d747699c95b8862473f985 WatchSource:0}: Error finding container e999b181aaec7248e4ee3ac9fd391d61a4a0ef65d1d747699c95b8862473f985: Status 404 returned error can't find the container with id e999b181aaec7248e4ee3ac9fd391d61a4a0ef65d1d747699c95b8862473f985
Sep 07 11:35:41.773509 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:41.773452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6677b4689b-sx8j5" event={"ID":"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6","Type":"ContainerStarted","Data":"e999b181aaec7248e4ee3ac9fd391d61a4a0ef65d1d747699c95b8862473f985"}
Sep 07 11:35:42.781636 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:42.781600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6677b4689b-sx8j5" event={"ID":"9eac05d8-8987-4b06-ada2-9d3c9de2c0b6","Type":"ContainerStarted","Data":"b28718a1845cf639f16fdfe776d406360c9e3817318df96f2e41436975f96510"}
Sep 07 11:35:42.800575 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:42.800515    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6677b4689b-sx8j5" podStartSLOduration=1.172196668 podStartE2EDuration="1.800494345s" totalImagesPullingTime="628.297677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:35:42.799715731 +0000 UTC m=+774.274563882" watchObservedRunningTime="2026-09-07 11:35:42.800494345 +0000 UTC m=+774.275342493"
Sep 07 11:35:42.917740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:42.917675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:43.345267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:43.345234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:43.441417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:43.441382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:44.477176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:44.477140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:44.575675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:44.575632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:49.053612 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:49.053571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:49.118704 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:49.118670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:35:51.343853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:35:51.343820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:00.978788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:00.978750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:01.082179 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:01.082147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:01.312055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:01.311972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:01.411744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:01.411706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:02.401728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:02.401688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:02.505169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:02.505137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:02.857594 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:02.857521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:02.944584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:02.944549    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:02.948252 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:02.948235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:02.950410 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:02.950388    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhmxq\""
Sep 07 11:36:02.961740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:02.961696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:03.051919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:03.051888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7vv\" (UniqueName: \"kubernetes.io/projected/8244d547-4326-40b2-aac1-f19674ac6810-kube-api-access-lg7vv\") pod \"test-curl-7233-139705534831408\" (UID: \"8244d547-4326-40b2-aac1-f19674ac6810\") " pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:03.152399 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:03.152359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7vv\" (UniqueName: \"kubernetes.io/projected/8244d547-4326-40b2-aac1-f19674ac6810-kube-api-access-lg7vv\") pod \"test-curl-7233-139705534831408\" (UID: \"8244d547-4326-40b2-aac1-f19674ac6810\") " pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:03.160872 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:03.160843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7vv\" (UniqueName: \"kubernetes.io/projected/8244d547-4326-40b2-aac1-f19674ac6810-kube-api-access-lg7vv\") pod \"test-curl-7233-139705534831408\" (UID: \"8244d547-4326-40b2-aac1-f19674ac6810\") " pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:03.258867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:03.258826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:03.528455 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:03.528428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:03.530031 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:03.529998    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8244d547_4326_40b2_aac1_f19674ac6810.slice/crio-0e73973080d178d4d10d4a3b513ecdf094f39cbfa81a043a1c8d6338d400d499 WatchSource:0}: Error finding container 0e73973080d178d4d10d4a3b513ecdf094f39cbfa81a043a1c8d6338d400d499: Status 404 returned error can't find the container with id 0e73973080d178d4d10d4a3b513ecdf094f39cbfa81a043a1c8d6338d400d499
Sep 07 11:36:03.889695 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:03.889662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7233-139705534831408" event={"ID":"8244d547-4326-40b2-aac1-f19674ac6810","Type":"ContainerStarted","Data":"0e73973080d178d4d10d4a3b513ecdf094f39cbfa81a043a1c8d6338d400d499"}
Sep 07 11:36:03.997679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:03.997640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:04.098180 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:04.098138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:04.598492 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:04.598436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:36:05.151979 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:05.151944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:05.226196 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:05.226157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:05.726282 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:05.726191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:05.824120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:05.824089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:05.898322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:05.898290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:05.903856 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:05.903829    2578 generic.go:356] "Generic (PLEG): container finished" podID="8244d547-4326-40b2-aac1-f19674ac6810" containerID="39137d40f350f23f5c31f3436a82d91d01219f5ed6073e07d9cfee8d7f47e45c" exitCode=0
Sep 07 11:36:05.903992 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:05.903877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7233-139705534831408" event={"ID":"8244d547-4326-40b2-aac1-f19674ac6810","Type":"ContainerDied","Data":"39137d40f350f23f5c31f3436a82d91d01219f5ed6073e07d9cfee8d7f47e45c"}
Sep 07 11:36:07.047079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.047051    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:07.066714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.066683    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7233-139705534831408_8244d547-4326-40b2-aac1-f19674ac6810/test-curl-7233-139705534831408/0.log"
Sep 07 11:36:07.085942 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.085835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg7vv\" (UniqueName: \"kubernetes.io/projected/8244d547-4326-40b2-aac1-f19674ac6810-kube-api-access-lg7vv\") pod \"8244d547-4326-40b2-aac1-f19674ac6810\" (UID: \"8244d547-4326-40b2-aac1-f19674ac6810\") "
Sep 07 11:36:07.088617 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.088582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8244d547-4326-40b2-aac1-f19674ac6810-kube-api-access-lg7vv" (OuterVolumeSpecName: "kube-api-access-lg7vv") pod "8244d547-4326-40b2-aac1-f19674ac6810" (UID: "8244d547-4326-40b2-aac1-f19674ac6810"). InnerVolumeSpecName "kube-api-access-lg7vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:36:07.089386 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.089367    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:07.091339 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.091319    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:07.187714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.187637    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg7vv\" (UniqueName: \"kubernetes.io/projected/8244d547-4326-40b2-aac1-f19674ac6810-kube-api-access-lg7vv\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:36:07.200512 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.200479    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:07.200813 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.200800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8244d547-4326-40b2-aac1-f19674ac6810" containerName="test-curl-7233-139705534831408"
Sep 07 11:36:07.200856 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.200815    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8244d547-4326-40b2-aac1-f19674ac6810" containerName="test-curl-7233-139705534831408"
Sep 07 11:36:07.200911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.200901    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8244d547-4326-40b2-aac1-f19674ac6810" containerName="test-curl-7233-139705534831408"
Sep 07 11:36:07.204023 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.204007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:07.213865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.213841    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:07.288197 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.288154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqdxr\" (UniqueName: \"kubernetes.io/projected/da6ee7ff-6dfa-4ed7-a115-5567f8df807c-kube-api-access-xqdxr\") pod \"test-curl-7233-139705534831408\" (UID: \"da6ee7ff-6dfa-4ed7-a115-5567f8df807c\") " pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:07.389349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.389304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqdxr\" (UniqueName: \"kubernetes.io/projected/da6ee7ff-6dfa-4ed7-a115-5567f8df807c-kube-api-access-xqdxr\") pod \"test-curl-7233-139705534831408\" (UID: \"da6ee7ff-6dfa-4ed7-a115-5567f8df807c\") " pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:07.400050 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.400024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqdxr\" (UniqueName: \"kubernetes.io/projected/da6ee7ff-6dfa-4ed7-a115-5567f8df807c-kube-api-access-xqdxr\") pod \"test-curl-7233-139705534831408\" (UID: \"da6ee7ff-6dfa-4ed7-a115-5567f8df807c\") " pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:07.513329 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.513240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:07.783785 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.783760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:07.786281 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:07.786256    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda6ee7ff_6dfa_4ed7_a115_5567f8df807c.slice/crio-7d0f6d79685bf675b5d0241bb3052844c4f978d7b0c4f04dd5fb47861a68a925 WatchSource:0}: Error finding container 7d0f6d79685bf675b5d0241bb3052844c4f978d7b0c4f04dd5fb47861a68a925: Status 404 returned error can't find the container with id 7d0f6d79685bf675b5d0241bb3052844c4f978d7b0c4f04dd5fb47861a68a925
Sep 07 11:36:07.915531 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.915492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7233-139705534831408" event={"ID":"da6ee7ff-6dfa-4ed7-a115-5567f8df807c","Type":"ContainerStarted","Data":"7d0f6d79685bf675b5d0241bb3052844c4f978d7b0c4f04dd5fb47861a68a925"}
Sep 07 11:36:07.916631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.916608    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:07.916753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.916647    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e73973080d178d4d10d4a3b513ecdf094f39cbfa81a043a1c8d6338d400d499"
Sep 07 11:36:07.928696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:07.928664    2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7233-139705534831408" oldPodUID="8244d547-4326-40b2-aac1-f19674ac6810" podUID="da6ee7ff-6dfa-4ed7-a115-5567f8df807c"
Sep 07 11:36:08.923197 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:08.923164    2578 generic.go:356] "Generic (PLEG): container finished" podID="da6ee7ff-6dfa-4ed7-a115-5567f8df807c" containerID="6af62c3753ffa495101b1215408ab187fa2336f484b151592bb6b7bdd9eaf010" exitCode=0
Sep 07 11:36:08.923611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:08.923256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7233-139705534831408" event={"ID":"da6ee7ff-6dfa-4ed7-a115-5567f8df807c","Type":"ContainerDied","Data":"6af62c3753ffa495101b1215408ab187fa2336f484b151592bb6b7bdd9eaf010"}
Sep 07 11:36:08.959628 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:08.959602    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8244d547-4326-40b2-aac1-f19674ac6810" path="/var/lib/kubelet/pods/8244d547-4326-40b2-aac1-f19674ac6810/volumes"
Sep 07 11:36:09.078189 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:09.078162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:09.141819 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:09.141790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:09.334616 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:09.334537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:09.627215 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:09.627184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:09.802848 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:09.802819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:10.076939 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.076915    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:10.098198 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.098123    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7233-139705534831408_da6ee7ff-6dfa-4ed7-a115-5567f8df807c/test-curl-7233-139705534831408/0.log"
Sep 07 11:36:10.110972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.110945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqdxr\" (UniqueName: \"kubernetes.io/projected/da6ee7ff-6dfa-4ed7-a115-5567f8df807c-kube-api-access-xqdxr\") pod \"da6ee7ff-6dfa-4ed7-a115-5567f8df807c\" (UID: \"da6ee7ff-6dfa-4ed7-a115-5567f8df807c\") "
Sep 07 11:36:10.113045 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.113022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6ee7ff-6dfa-4ed7-a115-5567f8df807c-kube-api-access-xqdxr" (OuterVolumeSpecName: "kube-api-access-xqdxr") pod "da6ee7ff-6dfa-4ed7-a115-5567f8df807c" (UID: "da6ee7ff-6dfa-4ed7-a115-5567f8df807c"). InnerVolumeSpecName "kube-api-access-xqdxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:36:10.123456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.123426    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:10.127138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.127117    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7233-139705534831408"]
Sep 07 11:36:10.211560 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.211489    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqdxr\" (UniqueName: \"kubernetes.io/projected/da6ee7ff-6dfa-4ed7-a115-5567f8df807c-kube-api-access-xqdxr\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:36:10.695534 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.695498    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:10.937758 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.937728    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d0f6d79685bf675b5d0241bb3052844c4f978d7b0c4f04dd5fb47861a68a925"
Sep 07 11:36:10.937758 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.937764    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:10.948236 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:36:10.948205    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7233-139705534831408\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="da6ee7ff-6dfa-4ed7-a115-5567f8df807c" pod="opendatahub/test-curl-7233-139705534831408"
Sep 07 11:36:10.960949 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.960917    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6ee7ff-6dfa-4ed7-a115-5567f8df807c" path="/var/lib/kubelet/pods/da6ee7ff-6dfa-4ed7-a115-5567f8df807c/volumes"
Sep 07 11:36:10.968812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.968780    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"]
Sep 07 11:36:10.969335 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.969314    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6ee7ff-6dfa-4ed7-a115-5567f8df807c" containerName="test-curl-7233-139705534831408"
Sep 07 11:36:10.969405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.969339    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6ee7ff-6dfa-4ed7-a115-5567f8df807c" containerName="test-curl-7233-139705534831408"
Sep 07 11:36:10.969485 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.969448    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6ee7ff-6dfa-4ed7-a115-5567f8df807c" containerName="test-curl-7233-139705534831408"
Sep 07 11:36:10.973044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.973021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:10.975402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.975382    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 11:36:10.975626 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.975539    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-152c50-openshift-default-dockercfg-pkqjs\""
Sep 07 11:36:10.983071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:10.983048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"]
Sep 07 11:36:11.001514 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.001486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:11.018037 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.017999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bhc\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-kube-api-access-q6bhc\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018037 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-certs\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-podinfo\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb07e9-6f5e-406c-9072-a58221b680c6-istiod-ca-cert\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-socket\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-token\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-envoy\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-credential-socket\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.018574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.018252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-data\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.118929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.118891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-podinfo\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.118929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.118932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb07e9-6f5e-406c-9072-a58221b680c6-istiod-ca-cert\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.118954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-socket\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.118978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-token\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.118997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-envoy\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-credential-socket\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-data\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bhc\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-kube-api-access-q6bhc\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-certs\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-socket\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119937 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-certs\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119937 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-data\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119937 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-credential-socket\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.119937 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.119789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb07e9-6f5e-406c-9072-a58221b680c6-istiod-ca-cert\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.121308 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.121287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-envoy\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.121621 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.121600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-podinfo\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.131165 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.131136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-token\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.131373 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.131354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bhc\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-kube-api-access-q6bhc\") pod \"e2e-autores-152c50-openshift-default-8479f4444b-mgbt8\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") " pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.192715 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.192678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:11.287141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.287064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:11.565704 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.565679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"]
Sep 07 11:36:11.567120 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:11.567096    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cb07e9_6f5e_406c_9072_a58221b680c6.slice/crio-5cf667ae20952a2192af33939f9a15a32ab93483ecd88aaa12e9c0efaea18049 WatchSource:0}: Error finding container 5cf667ae20952a2192af33939f9a15a32ab93483ecd88aaa12e9c0efaea18049: Status 404 returned error can't find the container with id 5cf667ae20952a2192af33939f9a15a32ab93483ecd88aaa12e9c0efaea18049
Sep 07 11:36:11.944661 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:11.944628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" event={"ID":"96cb07e9-6f5e-406c-9072-a58221b680c6","Type":"ContainerStarted","Data":"5cf667ae20952a2192af33939f9a15a32ab93483ecd88aaa12e9c0efaea18049"}
Sep 07 11:36:12.522919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:12.522867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:12.614079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:12.614036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:13.755430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:13.755397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:13.819403 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:13.819367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:14.025874 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.025785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:14.137625 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.137584    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:14.137740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.137662    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:14.137740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.137696    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:14.223798 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.223770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:14.315399 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.315311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:14.322981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.322952    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"]
Sep 07 11:36:14.327162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.327143    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.329786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.329765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c30b7add-dockercfg-vwk5z\""
Sep 07 11:36:14.329886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.329790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c30b7add\""
Sep 07 11:36:14.337191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.337168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"]
Sep 07 11:36:14.349760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.349703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.349856 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.349782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2wk7\" (UniqueName: \"kubernetes.io/projected/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-kube-api-access-b2wk7\") pod \"payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.450433 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.450400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.450609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.450441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2wk7\" (UniqueName: \"kubernetes.io/projected/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-kube-api-access-b2wk7\") pod \"payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.451331 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.451301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.458916 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.458884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2wk7\" (UniqueName: \"kubernetes.io/projected/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-kube-api-access-b2wk7\") pod \"payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.638955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.638922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:14.916817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.916792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"]
Sep 07 11:36:14.918648 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:14.918614    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df96b5c_2686_4b3f_97f7_4970b8ea50c0.slice/crio-e7a9d29c8640c77760b3f90ed376f4133038f3e935cc5798b8208c583f04e597 WatchSource:0}: Error finding container e7a9d29c8640c77760b3f90ed376f4133038f3e935cc5798b8208c583f04e597: Status 404 returned error can't find the container with id e7a9d29c8640c77760b3f90ed376f4133038f3e935cc5798b8208c583f04e597
Sep 07 11:36:14.973526 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.973488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" event={"ID":"96cb07e9-6f5e-406c-9072-a58221b680c6","Type":"ContainerStarted","Data":"5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83"}
Sep 07 11:36:14.974654 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.974630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl" event={"ID":"5df96b5c-2686-4b3f-97f7-4970b8ea50c0","Type":"ContainerStarted","Data":"e7a9d29c8640c77760b3f90ed376f4133038f3e935cc5798b8208c583f04e597"}
Sep 07 11:36:14.993366 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:14.993322    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podStartSLOduration=2.425353559 podStartE2EDuration="4.993307944s" totalImagesPullingTime="2.567954385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:14.991547469 +0000 UTC m=+806.466395641" watchObservedRunningTime="2026-09-07 11:36:14.993307944 +0000 UTC m=+806.468156094"
Sep 07 11:36:15.193044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:15.192966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:36:15.287830 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:15.287796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:15.292744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:15.292719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:15.982042 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:15.982015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:15.983269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:15.983245    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:16.926191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:16.926147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:16.989663 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:16.989621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl" event={"ID":"5df96b5c-2686-4b3f-97f7-4970b8ea50c0","Type":"ContainerStarted","Data":"8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775"}
Sep 07 11:36:16.990134 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:16.989932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:17.011099 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:17.011046    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl" podStartSLOduration=1.409699845 podStartE2EDuration="3.011031531s" totalImagesPullingTime="1.601331686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:17.00883447 +0000 UTC m=+808.483682639" watchObservedRunningTime="2026-09-07 11:36:17.011031531 +0000 UTC m=+808.485879679"
Sep 07 11:36:17.014729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:17.014706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:18.145957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.145918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:18.204388 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.204357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:18.665163 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.665129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:18.765369 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.765332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:18.855903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.855869    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"]
Sep 07 11:36:18.859316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.859298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:18.862027 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.862009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-152c50-dockercfg-kh8cr\""
Sep 07 11:36:18.862124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.862013    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-152c50\""
Sep 07 11:36:18.869009 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.868984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"]
Sep 07 11:36:18.890876 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.890853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6c951a-0e38-475a-9959-394665f287b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:18.891005 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.890909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59r5z\" (UniqueName: \"kubernetes.io/projected/1e6c951a-0e38-475a-9959-394665f287b2-kube-api-access-59r5z\") pod \"payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:18.949211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.949138    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"]
Sep 07 11:36:18.953241 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.953224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:18.970272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.970244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"]
Sep 07 11:36:18.991744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.991707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6c951a-0e38-475a-9959-394665f287b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:18.991905 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.991762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-plugins-config-volume\") pod \"payload-processing-e2e-autores-152c50-865455876f-ckzmm\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") " pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:18.991905 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.991813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59r5z\" (UniqueName: \"kubernetes.io/projected/1e6c951a-0e38-475a-9959-394665f287b2-kube-api-access-59r5z\") pod \"payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:18.991905 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.991898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnzn\" (UniqueName: \"kubernetes.io/projected/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-kube-api-access-gvnzn\") pod \"payload-processing-e2e-autores-152c50-865455876f-ckzmm\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") " pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:18.992485 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:18.992445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6c951a-0e38-475a-9959-394665f287b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:19.001556 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.001531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59r5z\" (UniqueName: \"kubernetes.io/projected/1e6c951a-0e38-475a-9959-394665f287b2-kube-api-access-59r5z\") pod \"payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:19.092648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.092608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnzn\" (UniqueName: \"kubernetes.io/projected/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-kube-api-access-gvnzn\") pod \"payload-processing-e2e-autores-152c50-865455876f-ckzmm\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") " pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:19.092792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.092682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-plugins-config-volume\") pod \"payload-processing-e2e-autores-152c50-865455876f-ckzmm\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") " pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:19.093371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.093349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-plugins-config-volume\") pod \"payload-processing-e2e-autores-152c50-865455876f-ckzmm\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") " pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:19.104052 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.104026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnzn\" (UniqueName: \"kubernetes.io/projected/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-kube-api-access-gvnzn\") pod \"payload-processing-e2e-autores-152c50-865455876f-ckzmm\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") " pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:19.169950 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.169926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:19.271222 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.271195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:19.441709 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.441686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"]
Sep 07 11:36:19.443532 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:19.443505    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6c951a_0e38_475a_9959_394665f287b2.slice/crio-35d06a472abbd92779782b51282914ffb20bfd52612085df154aa10596608515 WatchSource:0}: Error finding container 35d06a472abbd92779782b51282914ffb20bfd52612085df154aa10596608515: Status 404 returned error can't find the container with id 35d06a472abbd92779782b51282914ffb20bfd52612085df154aa10596608515
Sep 07 11:36:19.549041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.549017    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"]
Sep 07 11:36:19.550503 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:19.550477    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77dd24b_8110_4a06_8bab_95ef6f84e1ea.slice/crio-72b2ce22c535c1f55335ea6ba8be80736020eac72b6aec339d2510984c8ae8d1 WatchSource:0}: Error finding container 72b2ce22c535c1f55335ea6ba8be80736020eac72b6aec339d2510984c8ae8d1: Status 404 returned error can't find the container with id 72b2ce22c535c1f55335ea6ba8be80736020eac72b6aec339d2510984c8ae8d1
Sep 07 11:36:19.776296 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.776214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:19.936776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:19.936741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:20.010216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.010164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg" event={"ID":"1e6c951a-0e38-475a-9959-394665f287b2","Type":"ContainerStarted","Data":"dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6"}
Sep 07 11:36:20.010371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.010237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg" event={"ID":"1e6c951a-0e38-475a-9959-394665f287b2","Type":"ContainerStarted","Data":"35d06a472abbd92779782b51282914ffb20bfd52612085df154aa10596608515"}
Sep 07 11:36:20.010456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.010411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:20.012233 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.012208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm" event={"ID":"e77dd24b-8110-4a06-8bab-95ef6f84e1ea","Type":"ContainerStarted","Data":"e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf"}
Sep 07 11:36:20.012379 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.012256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm" event={"ID":"e77dd24b-8110-4a06-8bab-95ef6f84e1ea","Type":"ContainerStarted","Data":"72b2ce22c535c1f55335ea6ba8be80736020eac72b6aec339d2510984c8ae8d1"}
Sep 07 11:36:20.012379 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.012284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:20.037021 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.036941    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg" podStartSLOduration=2.036927882 podStartE2EDuration="2.036927882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:20.035534779 +0000 UTC m=+811.510382954" watchObservedRunningTime="2026-09-07 11:36:20.036927882 +0000 UTC m=+811.511776032"
Sep 07 11:36:20.041254 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.041230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:20.079028 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.078973    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm" podStartSLOduration=2.078956209 podStartE2EDuration="2.078956209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:20.075541147 +0000 UTC m=+811.550389316" watchObservedRunningTime="2026-09-07 11:36:20.078956209 +0000 UTC m=+811.553804363"
Sep 07 11:36:20.156848 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.156817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:20.247480 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.247433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:20.307878 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:20.307802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:22.857510 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:22.857457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:24.500176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:24.500142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:26.134721 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:26.134691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:27.104778 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:27.104744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:27.202053 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:27.202014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:27.292714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:27.292682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:27.462196 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:27.462166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:27.701492 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:27.701444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:36:27.996503 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:27.996452    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:28.962188 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:28.962153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:29.326225 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.326138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:29.623629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.623591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"]
Sep 07 11:36:29.631940 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.631904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.635436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.635407    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef61c5-openshift-default-dockercfg-txf7q\""
Sep 07 11:36:29.640017 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.639989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"]
Sep 07 11:36:29.652874 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.652840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:29.698026 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.697985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-certs\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-envoy\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-token\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istiod-ca-cert\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-credential-socket\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-socket\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-podinfo\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-data\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.698418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.698375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqbq\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-kube-api-access-6bqbq\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.799363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.799323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istiod-ca-cert\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.799592 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.799381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-credential-socket\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.799592 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.799446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-socket\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.799592 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.799523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-podinfo\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.799920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.799893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-credential-socket\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800008 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.799944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-socket\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800008 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.799952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-data\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.800025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqbq\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-kube-api-access-6bqbq\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.800088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-certs\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.800144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-envoy\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.800147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-data\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.800168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-token\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800422 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.800353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-certs\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.800422 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.800373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istiod-ca-cert\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.802425 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.802401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-envoy\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.802616 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.802517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-podinfo\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.808151 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.808123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-token\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.808340 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.808319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqbq\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-kube-api-access-6bqbq\") pod \"e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") " pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:29.945569 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:29.945458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:30.202253 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:30.202172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:30.255611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:30.255585    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"]
Sep 07 11:36:30.256888 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:30.256858    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7a5ef5_b785_4bf7_99a4_3559e7a6d7d7.slice/crio-eeea96128da6d54a58854737919379d7c6dd39cd2f3e8894b536d7b9fa350b0f WatchSource:0}: Error finding container eeea96128da6d54a58854737919379d7c6dd39cd2f3e8894b536d7b9fa350b0f: Status 404 returned error can't find the container with id eeea96128da6d54a58854737919379d7c6dd39cd2f3e8894b536d7b9fa350b0f
Sep 07 11:36:30.259300 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:30.259271    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:30.259356 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:30.259339    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:30.259399 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:30.259366    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:31.021676 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:31.021630    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:36:31.021888 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:31.021760    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:36:31.107349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:31.107307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" event={"ID":"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7","Type":"ContainerStarted","Data":"e277d4419979a00cf9b7fc299210022b0887703ebbd3f560029ad27c3ae62ee4"}
Sep 07 11:36:31.107514 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:31.107377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" event={"ID":"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7","Type":"ContainerStarted","Data":"eeea96128da6d54a58854737919379d7c6dd39cd2f3e8894b536d7b9fa350b0f"}
Sep 07 11:36:31.128404 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:31.128352    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" podStartSLOduration=2.128337632 podStartE2EDuration="2.128337632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:31.125999208 +0000 UTC m=+822.600847357" watchObservedRunningTime="2026-09-07 11:36:31.128337632 +0000 UTC m=+822.603185844"
Sep 07 11:36:31.946219 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:31.946177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:31.951373 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:31.951344    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:32.113058 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:32.113026    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:32.114080 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:32.114061    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:33.080299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:33.080247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:33.182197 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:33.182154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:33.897078 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:33.897041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:33.996795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:33.996764    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:34.098871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.098836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:34.257257 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.257179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:34.357494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.357448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:34.456518 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.456483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:34.821677 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.821639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:34.899560 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.899524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:34.928208 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.928173    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"]
Sep 07 11:36:34.935670 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.935643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:34.938706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.938683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ef61c5\""
Sep 07 11:36:34.942079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:34.942054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"]
Sep 07 11:36:35.053600 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.053435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9m9\" (UniqueName: \"kubernetes.io/projected/6d350635-d3ae-4820-86d2-476e05b079a7-kube-api-access-4m9m9\") pod \"payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:35.053600 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.053507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d350635-d3ae-4820-86d2-476e05b079a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:35.154940 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.154899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9m9\" (UniqueName: \"kubernetes.io/projected/6d350635-d3ae-4820-86d2-476e05b079a7-kube-api-access-4m9m9\") pod \"payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:35.155366 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.154956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d350635-d3ae-4820-86d2-476e05b079a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:35.155768 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.155747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d350635-d3ae-4820-86d2-476e05b079a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:35.163168 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.163133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9m9\" (UniqueName: \"kubernetes.io/projected/6d350635-d3ae-4820-86d2-476e05b079a7-kube-api-access-4m9m9\") pod \"payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:35.250622 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.250587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:35.319549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.319453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:35.527832 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:35.527806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"]
Sep 07 11:36:35.530699 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:35.530660    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d350635_d3ae_4820_86d2_476e05b079a7.slice/crio-bb64b4220774de7a7f2ca3334349d93812ee81214101b3a76c86dd75e1191e06 WatchSource:0}: Error finding container bb64b4220774de7a7f2ca3334349d93812ee81214101b3a76c86dd75e1191e06: Status 404 returned error can't find the container with id bb64b4220774de7a7f2ca3334349d93812ee81214101b3a76c86dd75e1191e06
Sep 07 11:36:36.036483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:36.036423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:36.137821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:36.137782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc" event={"ID":"6d350635-d3ae-4820-86d2-476e05b079a7","Type":"ContainerStarted","Data":"1e4823a73b501bc3c09cb1ba08a069c3cf22ad344b3a463e72676945f187c397"}
Sep 07 11:36:36.138001 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:36.137846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc" event={"ID":"6d350635-d3ae-4820-86d2-476e05b079a7","Type":"ContainerStarted","Data":"bb64b4220774de7a7f2ca3334349d93812ee81214101b3a76c86dd75e1191e06"}
Sep 07 11:36:36.138001 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:36.137971    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:36.156745 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:36.156693    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc" podStartSLOduration=2.156678998 podStartE2EDuration="2.156678998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:36.154239848 +0000 UTC m=+827.629088009" watchObservedRunningTime="2026-09-07 11:36:36.156678998 +0000 UTC m=+827.631527147"
Sep 07 11:36:36.708893 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:36.708865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:38.051027 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.050989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"]
Sep 07 11:36:38.053536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.053519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"
Sep 07 11:36:38.056136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.056118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:36:38.056215 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.056123    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:36:38.062549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.062523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"]
Sep 07 11:36:38.151882 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.151854    2578 generic.go:356] "Generic (PLEG): container finished" podID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerID="5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83" exitCode=0
Sep 07 11:36:38.152022 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.151937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" event={"ID":"96cb07e9-6f5e-406c-9072-a58221b680c6","Type":"ContainerDied","Data":"5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83"}
Sep 07 11:36:38.152315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.152299    2578 scope.go:117] "RemoveContainer" containerID="5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83"
Sep 07 11:36:38.153359 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.153332    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:38.153411 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.153396    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:38.153445 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.153431    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:38.183274 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.183248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8933e90f-ea26-4ded-89b3-d5a5273c5e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm\" (UID: \"8933e90f-ea26-4ded-89b3-d5a5273c5e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"
Sep 07 11:36:38.283935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.283900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8933e90f-ea26-4ded-89b3-d5a5273c5e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm\" (UID: \"8933e90f-ea26-4ded-89b3-d5a5273c5e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"
Sep 07 11:36:38.284624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.284596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8933e90f-ea26-4ded-89b3-d5a5273c5e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm\" (UID: \"8933e90f-ea26-4ded-89b3-d5a5273c5e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"
Sep 07 11:36:38.365284 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.365259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"
Sep 07 11:36:38.650816 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:36:38.650788    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8933e90f_ea26_4ded_89b3_d5a5273c5e3c.slice/crio-4d38bf88063bfee617e4bbd1a255ec2628ff1e5cfc4e33fe5bf8c36330c56ac5 WatchSource:0}: Error finding container 4d38bf88063bfee617e4bbd1a255ec2628ff1e5cfc4e33fe5bf8c36330c56ac5: Status 404 returned error can't find the container with id 4d38bf88063bfee617e4bbd1a255ec2628ff1e5cfc4e33fe5bf8c36330c56ac5
Sep 07 11:36:38.650959 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:38.650935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"]
Sep 07 11:36:39.159642 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:39.159597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" event={"ID":"96cb07e9-6f5e-406c-9072-a58221b680c6","Type":"ContainerStarted","Data":"353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9"}
Sep 07 11:36:39.160999 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:39.160971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm" event={"ID":"8933e90f-ea26-4ded-89b3-d5a5273c5e3c","Type":"ContainerStarted","Data":"4d38bf88063bfee617e4bbd1a255ec2628ff1e5cfc4e33fe5bf8c36330c56ac5"}
Sep 07 11:36:39.294963 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:39.294919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:39.295229 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:39.295048    2578 patch_prober.go:28] interesting pod/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 11:36:39.295229 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:39.295096    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 11:36:40.293390 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:40.292928    2578 patch_prober.go:28] interesting pod/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 11:36:40.293390 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:40.293007    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 11:36:40.562829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:40.562723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:41.287940 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:41.287512    2578 patch_prober.go:28] interesting pod/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 11:36:41.287940 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:41.287572    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 11:36:41.884969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:41.884936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:42.200939 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:42.200843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm" event={"ID":"8933e90f-ea26-4ded-89b3-d5a5273c5e3c","Type":"ContainerStarted","Data":"c96ab2095ac3b46b38b4cb4ba27c04e39100bd17abd84ab099c0a1bfa7f638ff"}
Sep 07 11:36:42.220970 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:42.220920    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm" podStartSLOduration=1.668924498 podStartE2EDuration="4.220904554s" totalImagesPullingTime="2.551980056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:42.217940379 +0000 UTC m=+833.692788550" watchObservedRunningTime="2026-09-07 11:36:42.220904554 +0000 UTC m=+833.695752702"
Sep 07 11:36:42.288218 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:42.287754    2578 patch_prober.go:28] interesting pod/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 11:36:42.288218 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:42.287819    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 11:36:42.902258 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:42.902202    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:36:43.210024 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:43.209930    2578 generic.go:356] "Generic (PLEG): container finished" podID="8933e90f-ea26-4ded-89b3-d5a5273c5e3c" containerID="c96ab2095ac3b46b38b4cb4ba27c04e39100bd17abd84ab099c0a1bfa7f638ff" exitCode=0
Sep 07 11:36:43.210024 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:43.210011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm" event={"ID":"8933e90f-ea26-4ded-89b3-d5a5273c5e3c","Type":"ContainerDied","Data":"c96ab2095ac3b46b38b4cb4ba27c04e39100bd17abd84ab099c0a1bfa7f638ff"}
Sep 07 11:36:43.290333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:43.290299    2578 patch_prober.go:28] interesting pod/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 11:36:43.290661 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:43.290618    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 11:36:44.287682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:44.287645    2578 patch_prober.go:28] interesting pod/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 11:36:44.288248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:44.287720    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 11:36:44.406006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:44.405974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"
Sep 07 11:36:44.548995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:44.548884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8933e90f-ea26-4ded-89b3-d5a5273c5e3c-maas-api-service-ca\") pod \"8933e90f-ea26-4ded-89b3-d5a5273c5e3c\" (UID: \"8933e90f-ea26-4ded-89b3-d5a5273c5e3c\") "
Sep 07 11:36:44.549343 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:44.549312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8933e90f-ea26-4ded-89b3-d5a5273c5e3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8933e90f-ea26-4ded-89b3-d5a5273c5e3c" (UID: "8933e90f-ea26-4ded-89b3-d5a5273c5e3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:36:44.650005 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:44.649962    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8933e90f-ea26-4ded-89b3-d5a5273c5e3c-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:36:45.231209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:45.231127    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"
Sep 07 11:36:45.231356 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:45.231123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm" event={"ID":"8933e90f-ea26-4ded-89b3-d5a5273c5e3c","Type":"ContainerDied","Data":"4d38bf88063bfee617e4bbd1a255ec2628ff1e5cfc4e33fe5bf8c36330c56ac5"}
Sep 07 11:36:45.231356 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:45.231249    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d38bf88063bfee617e4bbd1a255ec2628ff1e5cfc4e33fe5bf8c36330c56ac5"
Sep 07 11:36:45.295772 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:45.295741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:45.296131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:45.296033    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:45.296903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:45.296880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:36:47.147558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:47.147528    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:36:49.829189 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:49.829148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:53.279633 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:53.279547    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerID="e277d4419979a00cf9b7fc299210022b0887703ebbd3f560029ad27c3ae62ee4" exitCode=0
Sep 07 11:36:53.279633 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:53.279616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" event={"ID":"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7","Type":"ContainerDied","Data":"e277d4419979a00cf9b7fc299210022b0887703ebbd3f560029ad27c3ae62ee4"}
Sep 07 11:36:53.280069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:53.279968    2578 scope.go:117] "RemoveContainer" containerID="e277d4419979a00cf9b7fc299210022b0887703ebbd3f560029ad27c3ae62ee4"
Sep 07 11:36:53.281030 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:53.280999    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:53.281086 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:53.281076    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:53.281125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:53.281110    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:36:53.837516 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:53.837479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:54.292095 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.292049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" event={"ID":"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7","Type":"ContainerStarted","Data":"02e87caadb8a413d0b19bfc5153ef4bf2636458736495a423e75360806eb8c31"}
Sep 07 11:36:54.507252 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.507213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:54.535202 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.535165    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"]
Sep 07 11:36:54.535428 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.535397    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl" podUID="5df96b5c-2686-4b3f-97f7-4970b8ea50c0" containerName="payload-pre-processing" containerID="cri-o://8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775" gracePeriod=30
Sep 07 11:36:54.596305 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.595123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:54.695071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.695034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:54.695811 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.695346    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"]
Sep 07 11:36:54.699295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.699268    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c30b7add-jw5jm"]
Sep 07 11:36:54.794142 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.792824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:54.833822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.833792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:54.861069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.861024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:54.945984 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.945950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:54.946204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.946181    2578 patch_prober.go:28] interesting pod/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 07 11:36:54.946263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.946233    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 07 11:36:54.959000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.958971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2wk7\" (UniqueName: \"kubernetes.io/projected/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-kube-api-access-b2wk7\") pod \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") "
Sep 07 11:36:54.959143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.959023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-plugins-config-volume\") pod \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\" (UID: \"5df96b5c-2686-4b3f-97f7-4970b8ea50c0\") "
Sep 07 11:36:54.959404 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.959380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5df96b5c-2686-4b3f-97f7-4970b8ea50c0" (UID: "5df96b5c-2686-4b3f-97f7-4970b8ea50c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:36:54.961236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.961212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-kube-api-access-b2wk7" (OuterVolumeSpecName: "kube-api-access-b2wk7") pod "5df96b5c-2686-4b3f-97f7-4970b8ea50c0" (UID: "5df96b5c-2686-4b3f-97f7-4970b8ea50c0"). InnerVolumeSpecName "kube-api-access-b2wk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:36:54.961309 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:54.961256    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8933e90f-ea26-4ded-89b3-d5a5273c5e3c" path="/var/lib/kubelet/pods/8933e90f-ea26-4ded-89b3-d5a5273c5e3c/volumes"
Sep 07 11:36:55.060250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.060166    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2wk7\" (UniqueName: \"kubernetes.io/projected/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-kube-api-access-b2wk7\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:36:55.060250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.060194    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5df96b5c-2686-4b3f-97f7-4970b8ea50c0-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:36:55.220738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.220700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:55.304060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.301953    2578 generic.go:356] "Generic (PLEG): container finished" podID="5df96b5c-2686-4b3f-97f7-4970b8ea50c0" containerID="8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775" exitCode=0
Sep 07 11:36:55.304060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.302905    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"
Sep 07 11:36:55.304060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.303314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl" event={"ID":"5df96b5c-2686-4b3f-97f7-4970b8ea50c0","Type":"ContainerDied","Data":"8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775"}
Sep 07 11:36:55.304060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.303362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl" event={"ID":"5df96b5c-2686-4b3f-97f7-4970b8ea50c0","Type":"ContainerDied","Data":"e7a9d29c8640c77760b3f90ed376f4133038f3e935cc5798b8208c583f04e597"}
Sep 07 11:36:55.304060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.303391    2578 scope.go:117] "RemoveContainer" containerID="8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775"
Sep 07 11:36:55.315926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.315906    2578 scope.go:117] "RemoveContainer" containerID="8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775"
Sep 07 11:36:55.316223 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:36:55.316199    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775\": container with ID starting with 8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775 not found: ID does not exist" containerID="8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775"
Sep 07 11:36:55.316324 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.316229    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775"} err="failed to get container status \"8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775\": rpc error: code = NotFound desc = could not find container \"8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775\": container with ID starting with 8ce88a3bf27d917dfd4a734d3182d17d73665c01d6089a4e5db6ab390d610775 not found: ID does not exist"
Sep 07 11:36:55.323080 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.323060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:55.324526 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.324506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"]
Sep 07 11:36:55.326892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.326871    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c30b7add-66d77958c8-2j8vl"]
Sep 07 11:36:55.422548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.422513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:55.527777 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.527733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:55.915240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.915206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:55.946790 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.946759    2578 patch_prober.go:28] interesting pod/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 07 11:36:55.946957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:55.946810    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 07 11:36:56.905649 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:56.905611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:36:56.919926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:56.919891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:56.946079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:56.946050    2578 patch_prober.go:28] interesting pod/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 07 11:36:56.946194 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:56.946109    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 07 11:36:56.960420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:56.960393    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5df96b5c-2686-4b3f-97f7-4970b8ea50c0" path="/var/lib/kubelet/pods/5df96b5c-2686-4b3f-97f7-4970b8ea50c0/volumes"
Sep 07 11:36:57.946804 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:57.946759    2578 patch_prober.go:28] interesting pod/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 07 11:36:57.947185 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:57.946823    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 07 11:36:58.141338 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:58.141312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:58.934813 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:58.934783    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:58.945796 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:58.945768    2578 patch_prober.go:28] interesting pod/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 07 11:36:58.945953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:58.945816    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 07 11:36:59.033844 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.033805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:59.138380 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.138344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:59.233833 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.233745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:59.332919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.332881    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:59.686429 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.686396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:59.757885 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.757857    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"]
Sep 07 11:36:59.758286 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.758273    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8933e90f-ea26-4ded-89b3-d5a5273c5e3c" containerName="revoke-keys"
Sep 07 11:36:59.758330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.758291    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8933e90f-ea26-4ded-89b3-d5a5273c5e3c" containerName="revoke-keys"
Sep 07 11:36:59.758330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.758312    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df96b5c-2686-4b3f-97f7-4970b8ea50c0" containerName="payload-pre-processing"
Sep 07 11:36:59.758330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.758317    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df96b5c-2686-4b3f-97f7-4970b8ea50c0" containerName="payload-pre-processing"
Sep 07 11:36:59.758423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.758385    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df96b5c-2686-4b3f-97f7-4970b8ea50c0" containerName="payload-pre-processing"
Sep 07 11:36:59.758423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.758393    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8933e90f-ea26-4ded-89b3-d5a5273c5e3c" containerName="revoke-keys"
Sep 07 11:36:59.762808 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.762788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:36:59.763865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.763835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:36:59.765508 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.765488    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-25494d09-dockercfg-g76pb\""
Sep 07 11:36:59.765698 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.765677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-25494d09\""
Sep 07 11:36:59.776929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.776905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"]
Sep 07 11:36:59.806204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.806179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64zr\" (UniqueName: \"kubernetes.io/projected/75553922-811f-4ab3-9561-462b90d4a4b4-kube-api-access-k64zr\") pod \"payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:36:59.806323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.806229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75553922-811f-4ab3-9561-462b90d4a4b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:36:59.907744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.907704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64zr\" (UniqueName: \"kubernetes.io/projected/75553922-811f-4ab3-9561-462b90d4a4b4-kube-api-access-k64zr\") pod \"payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:36:59.907932 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.907769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75553922-811f-4ab3-9561-462b90d4a4b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:36:59.908454 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.908435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75553922-811f-4ab3-9561-462b90d4a4b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:36:59.920831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.920807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64zr\" (UniqueName: \"kubernetes.io/projected/75553922-811f-4ab3-9561-462b90d4a4b4-kube-api-access-k64zr\") pod \"payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:36:59.950489 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.950423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:59.950734 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.950717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:36:59.951489 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:36:59.951454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:37:00.073396 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:00.073365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:37:00.359861 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:00.359836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"]
Sep 07 11:37:00.361398 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:37:00.361369    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75553922_811f_4ab3_9561_462b90d4a4b4.slice/crio-1102188724c9c24b54372e323131658ac984ec3f2b7c373244cbca05bff72d1a WatchSource:0}: Error finding container 1102188724c9c24b54372e323131658ac984ec3f2b7c373244cbca05bff72d1a: Status 404 returned error can't find the container with id 1102188724c9c24b54372e323131658ac984ec3f2b7c373244cbca05bff72d1a
Sep 07 11:37:01.344483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:01.339813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb" event={"ID":"75553922-811f-4ab3-9561-462b90d4a4b4","Type":"ContainerStarted","Data":"1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a"}
Sep 07 11:37:01.344483 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:01.339924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb" event={"ID":"75553922-811f-4ab3-9561-462b90d4a4b4","Type":"ContainerStarted","Data":"1102188724c9c24b54372e323131658ac984ec3f2b7c373244cbca05bff72d1a"}
Sep 07 11:37:01.361954 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:01.361904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb" podStartSLOduration=2.361890575 podStartE2EDuration="2.361890575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:37:01.35969333 +0000 UTC m=+852.834541506" watchObservedRunningTime="2026-09-07 11:37:01.361890575 +0000 UTC m=+852.836738724"
Sep 07 11:37:02.345896 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:02.345866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:37:13.353893 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:13.353861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:37:13.903070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:13.903034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:37:23.916253 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:23.916217    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"]
Sep 07 11:37:23.919860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:23.919843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"
Sep 07 11:37:23.922583 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:23.922563    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:37:23.922701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:23.922566    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:37:23.928078 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:23.927929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"]
Sep 07 11:37:24.020051 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:24.020021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-25494d09-twfr5\" (UID: \"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"
Sep 07 11:37:24.121264 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:24.121228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-25494d09-twfr5\" (UID: \"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"
Sep 07 11:37:24.121906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:24.121886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-25494d09-twfr5\" (UID: \"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"
Sep 07 11:37:24.231855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:24.231775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"
Sep 07 11:37:24.510352 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:24.510323    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"]
Sep 07 11:37:24.511693 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:37:24.511656    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4437a066_a775_4d9f_aaf2_6ffb1f7f1d8b.slice/crio-3c4d5a35bafaea0d2771f7d689d139464bdd73e7e36bba299ff2f3a4f5bd63d9 WatchSource:0}: Error finding container 3c4d5a35bafaea0d2771f7d689d139464bdd73e7e36bba299ff2f3a4f5bd63d9: Status 404 returned error can't find the container with id 3c4d5a35bafaea0d2771f7d689d139464bdd73e7e36bba299ff2f3a4f5bd63d9
Sep 07 11:37:25.475224 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:25.475185    2578 generic.go:356] "Generic (PLEG): container finished" podID="4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b" containerID="07c95552a4123db5c301bce1c04e345290fc73f88e609508a5c6bf1977e47b0e" exitCode=0
Sep 07 11:37:25.475587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:25.475274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5" event={"ID":"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b","Type":"ContainerDied","Data":"07c95552a4123db5c301bce1c04e345290fc73f88e609508a5c6bf1977e47b0e"}
Sep 07 11:37:25.475587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:25.475332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5" event={"ID":"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b","Type":"ContainerStarted","Data":"3c4d5a35bafaea0d2771f7d689d139464bdd73e7e36bba299ff2f3a4f5bd63d9"}
Sep 07 11:37:26.616178 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:26.616155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"
Sep 07 11:37:26.745724 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:26.745636    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b-maas-api-service-ca\") pod \"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b\" (UID: \"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b\") "
Sep 07 11:37:26.745982 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:26.745960    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b" (UID: "4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:37:26.846857 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:26.846808    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:37:27.338705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:27.338670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:27.487875 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:27.487844    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"
Sep 07 11:37:27.488040 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:27.487895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5" event={"ID":"4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b","Type":"ContainerDied","Data":"3c4d5a35bafaea0d2771f7d689d139464bdd73e7e36bba299ff2f3a4f5bd63d9"}
Sep 07 11:37:27.488040 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:27.487941    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c4d5a35bafaea0d2771f7d689d139464bdd73e7e36bba299ff2f3a4f5bd63d9"
Sep 07 11:37:29.114534 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:29.114499    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:37:30.787263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:30.787232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:30.840504 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:30.840448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"]
Sep 07 11:37:30.841109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:30.841076    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb" podUID="75553922-811f-4ab3-9561-462b90d4a4b4" containerName="payload-pre-processing" containerID="cri-o://1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a" gracePeriod=30
Sep 07 11:37:30.863894 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:30.863860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:30.923866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:30.923833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:31.005844 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.005809    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"]
Sep 07 11:37:31.007685 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.007658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:31.009726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.009701    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25494d09-twfr5"]
Sep 07 11:37:31.073947 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.073853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:31.112183 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.112156    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:37:31.188782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.188748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k64zr\" (UniqueName: \"kubernetes.io/projected/75553922-811f-4ab3-9561-462b90d4a4b4-kube-api-access-k64zr\") pod \"75553922-811f-4ab3-9561-462b90d4a4b4\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") "
Sep 07 11:37:31.188964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.188861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75553922-811f-4ab3-9561-462b90d4a4b4-plugins-config-volume\") pod \"75553922-811f-4ab3-9561-462b90d4a4b4\" (UID: \"75553922-811f-4ab3-9561-462b90d4a4b4\") "
Sep 07 11:37:31.189222 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.189199    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75553922-811f-4ab3-9561-462b90d4a4b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75553922-811f-4ab3-9561-462b90d4a4b4" (UID: "75553922-811f-4ab3-9561-462b90d4a4b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:37:31.190833 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.190798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75553922-811f-4ab3-9561-462b90d4a4b4-kube-api-access-k64zr" (OuterVolumeSpecName: "kube-api-access-k64zr") pod "75553922-811f-4ab3-9561-462b90d4a4b4" (UID: "75553922-811f-4ab3-9561-462b90d4a4b4"). InnerVolumeSpecName "kube-api-access-k64zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:37:31.195002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.194974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:31.289516 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.289445    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75553922-811f-4ab3-9561-462b90d4a4b4-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:37:31.289516 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.289509    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k64zr\" (UniqueName: \"kubernetes.io/projected/75553922-811f-4ab3-9561-462b90d4a4b4-kube-api-access-k64zr\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:37:31.510758 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.510719    2578 generic.go:356] "Generic (PLEG): container finished" podID="75553922-811f-4ab3-9561-462b90d4a4b4" containerID="1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a" exitCode=0
Sep 07 11:37:31.510953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.510765    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"
Sep 07 11:37:31.510953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.510807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb" event={"ID":"75553922-811f-4ab3-9561-462b90d4a4b4","Type":"ContainerDied","Data":"1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a"}
Sep 07 11:37:31.510953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.510866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb" event={"ID":"75553922-811f-4ab3-9561-462b90d4a4b4","Type":"ContainerDied","Data":"1102188724c9c24b54372e323131658ac984ec3f2b7c373244cbca05bff72d1a"}
Sep 07 11:37:31.510953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.510891    2578 scope.go:117] "RemoveContainer" containerID="1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a"
Sep 07 11:37:31.520419 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.520401    2578 scope.go:117] "RemoveContainer" containerID="1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a"
Sep 07 11:37:31.520690 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:37:31.520674    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a\": container with ID starting with 1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a not found: ID does not exist" containerID="1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a"
Sep 07 11:37:31.520742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.520699    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a"} err="failed to get container status \"1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a\": rpc error: code = NotFound desc = could not find container \"1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a\": container with ID starting with 1b6bc806cc4b03ee81e1266ded68a9a960b32e86f4c8859e7d8df1ca0f91479a not found: ID does not exist"
Sep 07 11:37:31.532696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.532671    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"]
Sep 07 11:37:31.536448 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.536423    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25494d09-798dc5bd78-9mtdb"]
Sep 07 11:37:31.607576 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.607547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:31.951062 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:31.951023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:32.152341 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:32.152301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:32.961393 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:32.961361    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b" path="/var/lib/kubelet/pods/4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b/volumes"
Sep 07 11:37:32.961782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:32.961751    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75553922-811f-4ab3-9561-462b90d4a4b4" path="/var/lib/kubelet/pods/75553922-811f-4ab3-9561-462b90d4a4b4/volumes"
Sep 07 11:37:34.935344 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:34.935312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:35.033217 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:35.033176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:36.111629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.111588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:36.211494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.211439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:36.362855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.362772    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:36.608860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.608823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:36.677641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.677561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:37:36.706164 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.706126    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"]
Sep 07 11:37:36.706609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.706592    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75553922-811f-4ab3-9561-462b90d4a4b4" containerName="payload-pre-processing"
Sep 07 11:37:36.706609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.706610    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75553922-811f-4ab3-9561-462b90d4a4b4" containerName="payload-pre-processing"
Sep 07 11:37:36.706766 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.706621    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b" containerName="revoke-keys"
Sep 07 11:37:36.706766 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.706630    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b" containerName="revoke-keys"
Sep 07 11:37:36.706868 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.706768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4437a066-a775-4d9f-aaf2-6ffb1f7f1d8b" containerName="revoke-keys"
Sep 07 11:37:36.706868 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.706785    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="75553922-811f-4ab3-9561-462b90d4a4b4" containerName="payload-pre-processing"
Sep 07 11:37:36.711065 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.711044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:36.713715 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.713694    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5e86f370\""
Sep 07 11:37:36.719678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.719653    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"]
Sep 07 11:37:36.731639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.731617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19ad4fe-3e09-46a5-9d88-d7427b580920-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:36.731639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.731645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcbnw\" (UniqueName: \"kubernetes.io/projected/f19ad4fe-3e09-46a5-9d88-d7427b580920-kube-api-access-rcbnw\") pod \"payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:36.832954 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.832921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19ad4fe-3e09-46a5-9d88-d7427b580920-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:36.832954 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.832957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcbnw\" (UniqueName: \"kubernetes.io/projected/f19ad4fe-3e09-46a5-9d88-d7427b580920-kube-api-access-rcbnw\") pod \"payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:36.833683 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.833659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19ad4fe-3e09-46a5-9d88-d7427b580920-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:36.841228 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:36.841197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcbnw\" (UniqueName: \"kubernetes.io/projected/f19ad4fe-3e09-46a5-9d88-d7427b580920-kube-api-access-rcbnw\") pod \"payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:37.046027 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:37.045932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:37.325771 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:37.325726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"]
Sep 07 11:37:37.327348 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:37:37.327306    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19ad4fe_3e09_46a5_9d88_d7427b580920.slice/crio-a26c2ce370632a3f160484a08e819755bd3384dc7f32af80225c14acc70eb6ad WatchSource:0}: Error finding container a26c2ce370632a3f160484a08e819755bd3384dc7f32af80225c14acc70eb6ad: Status 404 returned error can't find the container with id a26c2ce370632a3f160484a08e819755bd3384dc7f32af80225c14acc70eb6ad
Sep 07 11:37:37.552543 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:37.552503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj" event={"ID":"f19ad4fe-3e09-46a5-9d88-d7427b580920","Type":"ContainerStarted","Data":"df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753"}
Sep 07 11:37:37.552745 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:37.552572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj" event={"ID":"f19ad4fe-3e09-46a5-9d88-d7427b580920","Type":"ContainerStarted","Data":"a26c2ce370632a3f160484a08e819755bd3384dc7f32af80225c14acc70eb6ad"}
Sep 07 11:37:37.552745 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:37.552732    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:37.580008 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:37.579906    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj" podStartSLOduration=1.579887427 podStartE2EDuration="1.579887427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:37:37.575667622 +0000 UTC m=+889.050515796" watchObservedRunningTime="2026-09-07 11:37:37.579887427 +0000 UTC m=+889.054735582"
Sep 07 11:37:45.699385 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:45.699300    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:37:48.560396 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:48.560364    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:37:49.344814 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:49.344729    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:37:49.347060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:49.347037    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:37:49.349502 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:49.349477    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:37:49.351766 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:37:49.351747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:38:00.768313 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.768275    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"]
Sep 07 11:38:00.773134 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.773116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"
Sep 07 11:38:00.776385 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.776358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:38:00.776541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.776411    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:38:00.778971 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.778944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"]
Sep 07 11:38:00.854545 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.854513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a9b67-3b02-4725-b05e-ae23653a6188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5e86f370-g5594\" (UID: \"e80a9b67-3b02-4725-b05e-ae23653a6188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"
Sep 07 11:38:00.897479 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.897441    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"]
Sep 07 11:38:00.900955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.900934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:00.903524 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.903501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-152c50\"/\"openshift-service-ca.crt\""
Sep 07 11:38:00.903779 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.903757    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-152c50\"/\"kube-root-ca.crt\""
Sep 07 11:38:00.903910 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.903800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-152c50\"/\"autores-model-152c50-kserve-self-signed-certs\""
Sep 07 11:38:00.903910 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.903838    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-152c50\"/\"default-dockercfg-wlcbb\""
Sep 07 11:38:00.911125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.911104    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"]
Sep 07 11:38:00.960322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.960285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a9b67-3b02-4725-b05e-ae23653a6188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5e86f370-g5594\" (UID: \"e80a9b67-3b02-4725-b05e-ae23653a6188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"
Sep 07 11:38:00.961361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.961334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a9b67-3b02-4725-b05e-ae23653a6188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5e86f370-g5594\" (UID: \"e80a9b67-3b02-4725-b05e-ae23653a6188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"
Sep 07 11:38:00.986423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:00.986396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:01.061565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.061457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-model-cache\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.061565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.061527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-home\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.061565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.061553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njld2\" (UniqueName: \"kubernetes.io/projected/01c97596-ed76-4f32-adad-ccbe3032d13d-kube-api-access-njld2\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.061785 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.061634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c97596-ed76-4f32-adad-ccbe3032d13d-tls-certs\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.061785 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.061651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-dshm\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.061785 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.061681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-tmp-dir\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.085342 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.085308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"
Sep 07 11:38:01.163123 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-dshm\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163123 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-tmp-dir\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-model-cache\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-home\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njld2\" (UniqueName: \"kubernetes.io/projected/01c97596-ed76-4f32-adad-ccbe3032d13d-kube-api-access-njld2\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c97596-ed76-4f32-adad-ccbe3032d13d-tls-certs\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163646 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-tmp-dir\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-model-cache\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.163981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.163972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-home\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.166114 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.166089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-dshm\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.166440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.166407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c97596-ed76-4f32-adad-ccbe3032d13d-tls-certs\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.170857 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.170835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njld2\" (UniqueName: \"kubernetes.io/projected/01c97596-ed76-4f32-adad-ccbe3032d13d-kube-api-access-njld2\") pod \"autores-model-152c50-kserve-58bf8b5899-zhmtx\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") " pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.212754 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.212716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:01.359230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.359198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"]
Sep 07 11:38:01.360320 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:38:01.360293    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80a9b67_3b02_4725_b05e_ae23653a6188.slice/crio-dada2fefde57c6fc39805af76668443a12b1973d2d855b5b9e8eaae1aa70b4ad WatchSource:0}: Error finding container dada2fefde57c6fc39805af76668443a12b1973d2d855b5b9e8eaae1aa70b4ad: Status 404 returned error can't find the container with id dada2fefde57c6fc39805af76668443a12b1973d2d855b5b9e8eaae1aa70b4ad
Sep 07 11:38:01.489975 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.489951    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"]
Sep 07 11:38:01.491591 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:38:01.491565    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01c97596_ed76_4f32_adad_ccbe3032d13d.slice/crio-4c00f44bf1c6355b2080165dd3a6e47c09a762378448d60698db2338335ffe81 WatchSource:0}: Error finding container 4c00f44bf1c6355b2080165dd3a6e47c09a762378448d60698db2338335ffe81: Status 404 returned error can't find the container with id 4c00f44bf1c6355b2080165dd3a6e47c09a762378448d60698db2338335ffe81
Sep 07 11:38:01.697681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.697650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx" event={"ID":"01c97596-ed76-4f32-adad-ccbe3032d13d","Type":"ContainerStarted","Data":"6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79"}
Sep 07 11:38:01.697816 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.697697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx" event={"ID":"01c97596-ed76-4f32-adad-ccbe3032d13d","Type":"ContainerStarted","Data":"4c00f44bf1c6355b2080165dd3a6e47c09a762378448d60698db2338335ffe81"}
Sep 07 11:38:01.699225 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.699198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594" event={"ID":"e80a9b67-3b02-4725-b05e-ae23653a6188","Type":"ContainerStarted","Data":"208e937628bb455f46bea7a935d6eb340d2f805a178c307e33afde2d5eee8ca1"}
Sep 07 11:38:01.699318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.699251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594" event={"ID":"e80a9b67-3b02-4725-b05e-ae23653a6188","Type":"ContainerStarted","Data":"dada2fefde57c6fc39805af76668443a12b1973d2d855b5b9e8eaae1aa70b4ad"}
Sep 07 11:38:01.715625 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.715578    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx" podStartSLOduration=1.715562839 podStartE2EDuration="1.715562839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:01.713694984 +0000 UTC m=+913.188543134" watchObservedRunningTime="2026-09-07 11:38:01.715562839 +0000 UTC m=+913.190410988"
Sep 07 11:38:01.728291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:01.728227    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594" podStartSLOduration=1.728206892 podStartE2EDuration="1.728206892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:01.728188045 +0000 UTC m=+913.203036221" watchObservedRunningTime="2026-09-07 11:38:01.728206892 +0000 UTC m=+913.203055043"
Sep 07 11:38:02.296397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:02.296360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:38:02.706338 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:02.706300    2578 generic.go:356] "Generic (PLEG): container finished" podID="e80a9b67-3b02-4725-b05e-ae23653a6188" containerID="208e937628bb455f46bea7a935d6eb340d2f805a178c307e33afde2d5eee8ca1" exitCode=0
Sep 07 11:38:02.706541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:02.706368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594" event={"ID":"e80a9b67-3b02-4725-b05e-ae23653a6188","Type":"ContainerDied","Data":"208e937628bb455f46bea7a935d6eb340d2f805a178c307e33afde2d5eee8ca1"}
Sep 07 11:38:03.847696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:03.847671    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"
Sep 07 11:38:03.991491 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:03.991409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a9b67-3b02-4725-b05e-ae23653a6188-maas-api-service-ca\") pod \"e80a9b67-3b02-4725-b05e-ae23653a6188\" (UID: \"e80a9b67-3b02-4725-b05e-ae23653a6188\") "
Sep 07 11:38:03.991747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:03.991725    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80a9b67-3b02-4725-b05e-ae23653a6188-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e80a9b67-3b02-4725-b05e-ae23653a6188" (UID: "e80a9b67-3b02-4725-b05e-ae23653a6188"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:04.092105 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:04.092070    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a9b67-3b02-4725-b05e-ae23653a6188-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:04.720099 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:04.720069    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"
Sep 07 11:38:04.720278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:04.720071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594" event={"ID":"e80a9b67-3b02-4725-b05e-ae23653a6188","Type":"ContainerDied","Data":"dada2fefde57c6fc39805af76668443a12b1973d2d855b5b9e8eaae1aa70b4ad"}
Sep 07 11:38:04.720278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:04.720198    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dada2fefde57c6fc39805af76668443a12b1973d2d855b5b9e8eaae1aa70b4ad"
Sep 07 11:38:06.732985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.732953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:06.769612 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.769578    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"]
Sep 07 11:38:06.769822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.769795    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj" podUID="f19ad4fe-3e09-46a5-9d88-d7427b580920" containerName="payload-pre-processing" containerID="cri-o://df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753" gracePeriod=30
Sep 07 11:38:06.834337 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.834291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:06.943611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.943579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:06.945950 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.945921    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"]
Sep 07 11:38:06.948913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.948885    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e86f370-g5594"]
Sep 07 11:38:06.960525 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:06.960499    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80a9b67-3b02-4725-b05e-ae23653a6188" path="/var/lib/kubelet/pods/e80a9b67-3b02-4725-b05e-ae23653a6188/volumes"
Sep 07 11:38:07.027171 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.027144    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:38:07.035765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.035739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:07.120534 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.120493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19ad4fe-3e09-46a5-9d88-d7427b580920-plugins-config-volume\") pod \"f19ad4fe-3e09-46a5-9d88-d7427b580920\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") "
Sep 07 11:38:07.120716 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.120570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcbnw\" (UniqueName: \"kubernetes.io/projected/f19ad4fe-3e09-46a5-9d88-d7427b580920-kube-api-access-rcbnw\") pod \"f19ad4fe-3e09-46a5-9d88-d7427b580920\" (UID: \"f19ad4fe-3e09-46a5-9d88-d7427b580920\") "
Sep 07 11:38:07.120943 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.120917    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19ad4fe-3e09-46a5-9d88-d7427b580920-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f19ad4fe-3e09-46a5-9d88-d7427b580920" (UID: "f19ad4fe-3e09-46a5-9d88-d7427b580920"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:07.122645 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.122623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19ad4fe-3e09-46a5-9d88-d7427b580920-kube-api-access-rcbnw" (OuterVolumeSpecName: "kube-api-access-rcbnw") pod "f19ad4fe-3e09-46a5-9d88-d7427b580920" (UID: "f19ad4fe-3e09-46a5-9d88-d7427b580920"). InnerVolumeSpecName "kube-api-access-rcbnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:07.221618 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.221584    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19ad4fe-3e09-46a5-9d88-d7427b580920-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:07.221618 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.221613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcbnw\" (UniqueName: \"kubernetes.io/projected/f19ad4fe-3e09-46a5-9d88-d7427b580920-kube-api-access-rcbnw\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:07.744536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.744502    2578 generic.go:356] "Generic (PLEG): container finished" podID="f19ad4fe-3e09-46a5-9d88-d7427b580920" containerID="df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753" exitCode=0
Sep 07 11:38:07.744978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.744550    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"
Sep 07 11:38:07.744978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.744562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj" event={"ID":"f19ad4fe-3e09-46a5-9d88-d7427b580920","Type":"ContainerDied","Data":"df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753"}
Sep 07 11:38:07.744978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.744603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj" event={"ID":"f19ad4fe-3e09-46a5-9d88-d7427b580920","Type":"ContainerDied","Data":"a26c2ce370632a3f160484a08e819755bd3384dc7f32af80225c14acc70eb6ad"}
Sep 07 11:38:07.744978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.744625    2578 scope.go:117] "RemoveContainer" containerID="df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753"
Sep 07 11:38:07.754424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.754402    2578 scope.go:117] "RemoveContainer" containerID="df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753"
Sep 07 11:38:07.754664 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:38:07.754646    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753\": container with ID starting with df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753 not found: ID does not exist" containerID="df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753"
Sep 07 11:38:07.754713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.754671    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753"} err="failed to get container status \"df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753\": rpc error: code = NotFound desc = could not find container \"df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753\": container with ID starting with df0d3dfb1ee0f56e6141dbe162eebf43f8f79675521c9a15ffdf538ef578f753 not found: ID does not exist"
Sep 07 11:38:07.767598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.767574    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"]
Sep 07 11:38:07.771206 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:07.771183    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e86f370-584895977d-8dthj"]
Sep 07 11:38:08.280775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:08.280719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:08.717675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:08.717640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:08.960319 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:08.960287    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19ad4fe-3e09-46a5-9d88-d7427b580920" path="/var/lib/kubelet/pods/f19ad4fe-3e09-46a5-9d88-d7427b580920/volumes"
Sep 07 11:38:09.029580 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:09.029491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:09.129961 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:09.129915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:11.140681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:11.140647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:11.213179 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:11.213144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:11.213322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:11.213192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:11.225752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:11.225720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:11.238825 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:11.238795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:11.779523 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:11.779444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:12.436831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:12.436802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:12.536215 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:12.536175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:13.000452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.000417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:13.100937 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.100900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:13.193845 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.193809    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"]
Sep 07 11:38:13.194044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.193968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:13.194283 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.194264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80a9b67-3b02-4725-b05e-ae23653a6188" containerName="revoke-keys"
Sep 07 11:38:13.194350 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.194285    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80a9b67-3b02-4725-b05e-ae23653a6188" containerName="revoke-keys"
Sep 07 11:38:13.194350 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.194295    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19ad4fe-3e09-46a5-9d88-d7427b580920" containerName="payload-pre-processing"
Sep 07 11:38:13.194350 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.194300    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19ad4fe-3e09-46a5-9d88-d7427b580920" containerName="payload-pre-processing"
Sep 07 11:38:13.194530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.194383    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19ad4fe-3e09-46a5-9d88-d7427b580920" containerName="payload-pre-processing"
Sep 07 11:38:13.194530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.194414    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80a9b67-3b02-4725-b05e-ae23653a6188" containerName="revoke-keys"
Sep 07 11:38:13.199072 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.199046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.201934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.201903    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3f8729d3-dockercfg-n7w67\""
Sep 07 11:38:13.202125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.202105    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3f8729d3\""
Sep 07 11:38:13.211835 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.211811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"]
Sep 07 11:38:13.282788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.282705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4f6x\" (UniqueName: \"kubernetes.io/projected/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-kube-api-access-w4f6x\") pod \"payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.282788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.282751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.384020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.383987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4f6x\" (UniqueName: \"kubernetes.io/projected/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-kube-api-access-w4f6x\") pod \"payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.384191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.384027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.384796 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.384776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.393970 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.393948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4f6x\" (UniqueName: \"kubernetes.io/projected/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-kube-api-access-w4f6x\") pod \"payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.510148 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.510116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:13.785150 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:13.783846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"]
Sep 07 11:38:13.785336 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:38:13.785143    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee5eeb7_a486_4c68_8d8b_e8457aa580a1.slice/crio-d99cb751cd45ca54e167429d400f23585d227adf63a8640d565386610d27d621 WatchSource:0}: Error finding container d99cb751cd45ca54e167429d400f23585d227adf63a8640d565386610d27d621: Status 404 returned error can't find the container with id d99cb751cd45ca54e167429d400f23585d227adf63a8640d565386610d27d621
Sep 07 11:38:14.791148 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:14.791114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn" event={"ID":"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1","Type":"ContainerStarted","Data":"20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319"}
Sep 07 11:38:14.791609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:14.791165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn" event={"ID":"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1","Type":"ContainerStarted","Data":"d99cb751cd45ca54e167429d400f23585d227adf63a8640d565386610d27d621"}
Sep 07 11:38:14.791609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:14.791269    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:14.810723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:14.810677    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn" podStartSLOduration=1.810663226 podStartE2EDuration="1.810663226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:14.807752706 +0000 UTC m=+926.282600882" watchObservedRunningTime="2026-09-07 11:38:14.810663226 +0000 UTC m=+926.285511375"
Sep 07 11:38:22.407217 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:22.407179    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:38:25.799038 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:25.799002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:29.695342 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.695299    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"]
Sep 07 11:38:29.699618 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.699596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.703482 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.703438    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b7e4c1\"/\"kube-root-ca.crt\""
Sep 07 11:38:29.703620 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.703497    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b7e4c1\"/\"openshift-service-ca.crt\""
Sep 07 11:38:29.703620 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.703446    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b7e4c1\"/\"test-model-b7e4c1-kserve-self-signed-certs\""
Sep 07 11:38:29.703845 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.703824    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b7e4c1\"/\"default-dockercfg-cbls5\""
Sep 07 11:38:29.707092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.707068    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"]
Sep 07 11:38:29.720862 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.720823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:29.824205 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.824166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-home\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.824205 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.824207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhn7\" (UniqueName: \"kubernetes.io/projected/14d69976-10f6-47db-9cae-10327562842f-kube-api-access-tmhn7\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.824423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.824244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-tmp-dir\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.824423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.824300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d69976-10f6-47db-9cae-10327562842f-tls-certs\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.824423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.824342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-model-cache\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.824423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.824367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-dshm\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925313 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-model-cache\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-dshm\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-home\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhn7\" (UniqueName: \"kubernetes.io/projected/14d69976-10f6-47db-9cae-10327562842f-kube-api-access-tmhn7\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-tmp-dir\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d69976-10f6-47db-9cae-10327562842f-tls-certs\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-model-cache\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-home\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.925903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.925863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-tmp-dir\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.927622 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.927596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-dshm\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.927929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.927912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d69976-10f6-47db-9cae-10327562842f-tls-certs\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:29.934103 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:29.934085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhn7\" (UniqueName: \"kubernetes.io/projected/14d69976-10f6-47db-9cae-10327562842f-kube-api-access-tmhn7\") pod \"test-model-b7e4c1-kserve-f8c8dc869-dcljz\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") " pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:30.010723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:30.010645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:30.286242 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:30.286213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"]
Sep 07 11:38:30.287762 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:38:30.287736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14d69976_10f6_47db_9cae_10327562842f.slice/crio-87a05a6ddb0038e05dbba1842d8dff83b9fe7ac8a3dfaf83b8a4f8f46a918494 WatchSource:0}: Error finding container 87a05a6ddb0038e05dbba1842d8dff83b9fe7ac8a3dfaf83b8a4f8f46a918494: Status 404 returned error can't find the container with id 87a05a6ddb0038e05dbba1842d8dff83b9fe7ac8a3dfaf83b8a4f8f46a918494
Sep 07 11:38:30.886252 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:30.886211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz" event={"ID":"14d69976-10f6-47db-9cae-10327562842f","Type":"ContainerStarted","Data":"bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b"}
Sep 07 11:38:30.886714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:30.886262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz" event={"ID":"14d69976-10f6-47db-9cae-10327562842f","Type":"ContainerStarted","Data":"87a05a6ddb0038e05dbba1842d8dff83b9fe7ac8a3dfaf83b8a4f8f46a918494"}
Sep 07 11:38:30.903928 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:30.903874    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz" podStartSLOduration=1.9038603090000001 podStartE2EDuration="1.903860309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:30.902423626 +0000 UTC m=+942.377271770" watchObservedRunningTime="2026-09-07 11:38:30.903860309 +0000 UTC m=+942.378708458"
Sep 07 11:38:31.648761 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:31.648726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:32.015234 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:32.015160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:32.926081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:32.926054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:33.402913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.402883    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"]
Sep 07 11:38:33.415628 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.415601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:33.532087 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.532056    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"]
Sep 07 11:38:33.535764 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.535749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"
Sep 07 11:38:33.538399 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.538378    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:38:33.538571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.538553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:38:33.544653 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.544633    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"]
Sep 07 11:38:33.661667 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.661579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423cd5d8-017d-4d8b-83a2-b189a7b9c9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-152c50-jlzst\" (UID: \"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"
Sep 07 11:38:33.762333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.762292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423cd5d8-017d-4d8b-83a2-b189a7b9c9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-152c50-jlzst\" (UID: \"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"
Sep 07 11:38:33.762995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.762975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423cd5d8-017d-4d8b-83a2-b189a7b9c9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-152c50-jlzst\" (UID: \"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"
Sep 07 11:38:33.847677 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:33.847645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"
Sep 07 11:38:34.102760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:34.102682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:34.126114 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:34.126088    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"]
Sep 07 11:38:34.128276 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:38:34.128246    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423cd5d8_017d_4d8b_83a2_b189a7b9c9ed.slice/crio-76bbf1d258030abe8340d4eea0d42639642c22e03110db6181c015242796f81c WatchSource:0}: Error finding container 76bbf1d258030abe8340d4eea0d42639642c22e03110db6181c015242796f81c: Status 404 returned error can't find the container with id 76bbf1d258030abe8340d4eea0d42639642c22e03110db6181c015242796f81c
Sep 07 11:38:34.838408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:34.838371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:34.913941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:34.913905    2578 generic.go:356] "Generic (PLEG): container finished" podID="423cd5d8-017d-4d8b-83a2-b189a7b9c9ed" containerID="38db1fadd657bebdf82bb2e1d61dd5f479433399f0d3c22158a5334926a86008" exitCode=0
Sep 07 11:38:34.914108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:34.913971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst" event={"ID":"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed","Type":"ContainerDied","Data":"38db1fadd657bebdf82bb2e1d61dd5f479433399f0d3c22158a5334926a86008"}
Sep 07 11:38:34.914108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:34.914017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst" event={"ID":"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed","Type":"ContainerStarted","Data":"76bbf1d258030abe8340d4eea0d42639642c22e03110db6181c015242796f81c"}
Sep 07 11:38:36.054945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:36.054920    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"
Sep 07 11:38:36.184494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:36.184384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423cd5d8-017d-4d8b-83a2-b189a7b9c9ed-maas-api-service-ca\") pod \"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed\" (UID: \"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed\") "
Sep 07 11:38:36.184810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:36.184778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423cd5d8-017d-4d8b-83a2-b189a7b9c9ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "423cd5d8-017d-4d8b-83a2-b189a7b9c9ed" (UID: "423cd5d8-017d-4d8b-83a2-b189a7b9c9ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:36.285098 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:36.285069    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423cd5d8-017d-4d8b-83a2-b189a7b9c9ed-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:36.936450 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:36.936421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"
Sep 07 11:38:36.936450 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:36.936443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst" event={"ID":"423cd5d8-017d-4d8b-83a2-b189a7b9c9ed","Type":"ContainerDied","Data":"76bbf1d258030abe8340d4eea0d42639642c22e03110db6181c015242796f81c"}
Sep 07 11:38:36.936662 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:36.936520    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76bbf1d258030abe8340d4eea0d42639642c22e03110db6181c015242796f81c"
Sep 07 11:38:38.015190 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:38.015151    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:38:40.011240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:40.011208    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:40.011658 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:40.011256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:40.023833 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:40.023810    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:40.972882 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:40.972849    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:38:43.701074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:43.701042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:44.311159 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.311122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:44.332837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.332808    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"]
Sep 07 11:38:44.333051 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.333029    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm" podUID="e77dd24b-8110-4a06-8bab-95ef6f84e1ea" containerName="payload-processing" containerID="cri-o://e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf" gracePeriod=30
Sep 07 11:38:44.344295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.344269    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"]
Sep 07 11:38:44.344531 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.344504    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg" podUID="1e6c951a-0e38-475a-9959-394665f287b2" containerName="payload-pre-processing" containerID="cri-o://dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6" gracePeriod=30
Sep 07 11:38:44.406957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.406922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:44.512184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.512146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:44.516926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.516895    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"]
Sep 07 11:38:44.519360 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.519331    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-152c50-jlzst"]
Sep 07 11:38:44.608567 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.608536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:44.613783 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.613761    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:38:44.617116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.617092    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:38:44.770294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.770262    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvnzn\" (UniqueName: \"kubernetes.io/projected/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-kube-api-access-gvnzn\") pod \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") "
Sep 07 11:38:44.770294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.770299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6c951a-0e38-475a-9959-394665f287b2-plugins-config-volume\") pod \"1e6c951a-0e38-475a-9959-394665f287b2\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") "
Sep 07 11:38:44.770838 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.770417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59r5z\" (UniqueName: \"kubernetes.io/projected/1e6c951a-0e38-475a-9959-394665f287b2-kube-api-access-59r5z\") pod \"1e6c951a-0e38-475a-9959-394665f287b2\" (UID: \"1e6c951a-0e38-475a-9959-394665f287b2\") "
Sep 07 11:38:44.770838 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.770532    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-plugins-config-volume\") pod \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\" (UID: \"e77dd24b-8110-4a06-8bab-95ef6f84e1ea\") "
Sep 07 11:38:44.770838 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.770736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6c951a-0e38-475a-9959-394665f287b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e6c951a-0e38-475a-9959-394665f287b2" (UID: "1e6c951a-0e38-475a-9959-394665f287b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:44.770943 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.770848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e77dd24b-8110-4a06-8bab-95ef6f84e1ea" (UID: "e77dd24b-8110-4a06-8bab-95ef6f84e1ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:44.770943 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.770892    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6c951a-0e38-475a-9959-394665f287b2-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:44.772598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.772575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-kube-api-access-gvnzn" (OuterVolumeSpecName: "kube-api-access-gvnzn") pod "e77dd24b-8110-4a06-8bab-95ef6f84e1ea" (UID: "e77dd24b-8110-4a06-8bab-95ef6f84e1ea"). InnerVolumeSpecName "kube-api-access-gvnzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:44.772689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.772608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6c951a-0e38-475a-9959-394665f287b2-kube-api-access-59r5z" (OuterVolumeSpecName: "kube-api-access-59r5z") pod "1e6c951a-0e38-475a-9959-394665f287b2" (UID: "1e6c951a-0e38-475a-9959-394665f287b2"). InnerVolumeSpecName "kube-api-access-59r5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:44.860955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.860931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:44.872268 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.872243    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnzn\" (UniqueName: \"kubernetes.io/projected/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-kube-api-access-gvnzn\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:44.872268 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.872266    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59r5z\" (UniqueName: \"kubernetes.io/projected/1e6c951a-0e38-475a-9959-394665f287b2-kube-api-access-59r5z\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:44.872388 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.872276    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77dd24b-8110-4a06-8bab-95ef6f84e1ea-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:44.960783 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.960747    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423cd5d8-017d-4d8b-83a2-b189a7b9c9ed" path="/var/lib/kubelet/pods/423cd5d8-017d-4d8b-83a2-b189a7b9c9ed/volumes"
Sep 07 11:38:44.985347 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.985321    2578 generic.go:356] "Generic (PLEG): container finished" podID="e77dd24b-8110-4a06-8bab-95ef6f84e1ea" containerID="e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf" exitCode=0
Sep 07 11:38:44.985488 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.985374    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"
Sep 07 11:38:44.985488 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.985398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm" event={"ID":"e77dd24b-8110-4a06-8bab-95ef6f84e1ea","Type":"ContainerDied","Data":"e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf"}
Sep 07 11:38:44.985488 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.985450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm" event={"ID":"e77dd24b-8110-4a06-8bab-95ef6f84e1ea","Type":"ContainerDied","Data":"72b2ce22c535c1f55335ea6ba8be80736020eac72b6aec339d2510984c8ae8d1"}
Sep 07 11:38:44.985659 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.985490    2578 scope.go:117] "RemoveContainer" containerID="e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf"
Sep 07 11:38:44.986985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.986959    2578 generic.go:356] "Generic (PLEG): container finished" podID="1e6c951a-0e38-475a-9959-394665f287b2" containerID="dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6" exitCode=0
Sep 07 11:38:44.987101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.986997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg" event={"ID":"1e6c951a-0e38-475a-9959-394665f287b2","Type":"ContainerDied","Data":"dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6"}
Sep 07 11:38:44.987101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.987021    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"
Sep 07 11:38:44.987101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:44.987063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg" event={"ID":"1e6c951a-0e38-475a-9959-394665f287b2","Type":"ContainerDied","Data":"35d06a472abbd92779782b51282914ffb20bfd52612085df154aa10596608515"}
Sep 07 11:38:45.012145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.004747    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"]
Sep 07 11:38:45.012145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.010720    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-152c50-5d689bcdb-hv9kg"]
Sep 07 11:38:45.017796 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.017775    2578 scope.go:117] "RemoveContainer" containerID="e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf"
Sep 07 11:38:45.018158 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:38:45.018132    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf\": container with ID starting with e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf not found: ID does not exist" containerID="e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf"
Sep 07 11:38:45.018252 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.018167    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf"} err="failed to get container status \"e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf\": rpc error: code = NotFound desc = could not find container \"e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf\": container with ID starting with e6e74683c3bc6cec2f2b6d12835abfde4432ec97100f2e38e2cc70b4d38a0fcf not found: ID does not exist"
Sep 07 11:38:45.018252 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.018192    2578 scope.go:117] "RemoveContainer" containerID="dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6"
Sep 07 11:38:45.021729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.021707    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"]
Sep 07 11:38:45.025211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.025185    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-152c50-865455876f-ckzmm"]
Sep 07 11:38:45.028555 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.028538    2578 scope.go:117] "RemoveContainer" containerID="dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6"
Sep 07 11:38:45.028836 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:38:45.028817    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6\": container with ID starting with dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6 not found: ID does not exist" containerID="dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6"
Sep 07 11:38:45.028883 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:45.028844    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6"} err="failed to get container status \"dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6\": rpc error: code = NotFound desc = could not find container \"dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6\": container with ID starting with dc188332873aa1855ccfd3efa3c72820715e7365f67c7e3b47ca2c2b8c53a7b6 not found: ID does not exist"
Sep 07 11:38:46.960336 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:46.960303    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6c951a-0e38-475a-9959-394665f287b2" path="/var/lib/kubelet/pods/1e6c951a-0e38-475a-9959-394665f287b2/volumes"
Sep 07 11:38:46.960722 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:46.960688    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e77dd24b-8110-4a06-8bab-95ef6f84e1ea" path="/var/lib/kubelet/pods/e77dd24b-8110-4a06-8bab-95ef6f84e1ea/volumes"
Sep 07 11:38:48.423279 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.423230    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx" podUID="01c97596-ed76-4f32-adad-ccbe3032d13d" containerName="main" containerID="cri-o://6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79" gracePeriod=45
Sep 07 11:38:48.680758 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.680696    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:48.811057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njld2\" (UniqueName: \"kubernetes.io/projected/01c97596-ed76-4f32-adad-ccbe3032d13d-kube-api-access-njld2\") pod \"01c97596-ed76-4f32-adad-ccbe3032d13d\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") "
Sep 07 11:38:48.811216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-model-cache\") pod \"01c97596-ed76-4f32-adad-ccbe3032d13d\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") "
Sep 07 11:38:48.811216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c97596-ed76-4f32-adad-ccbe3032d13d-tls-certs\") pod \"01c97596-ed76-4f32-adad-ccbe3032d13d\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") "
Sep 07 11:38:48.811216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-dshm\") pod \"01c97596-ed76-4f32-adad-ccbe3032d13d\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") "
Sep 07 11:38:48.811216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-tmp-dir\") pod \"01c97596-ed76-4f32-adad-ccbe3032d13d\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") "
Sep 07 11:38:48.811438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-home\") pod \"01c97596-ed76-4f32-adad-ccbe3032d13d\" (UID: \"01c97596-ed76-4f32-adad-ccbe3032d13d\") "
Sep 07 11:38:48.811438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-model-cache" (OuterVolumeSpecName: "model-cache") pod "01c97596-ed76-4f32-adad-ccbe3032d13d" (UID: "01c97596-ed76-4f32-adad-ccbe3032d13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.811583 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-home" (OuterVolumeSpecName: "home") pod "01c97596-ed76-4f32-adad-ccbe3032d13d" (UID: "01c97596-ed76-4f32-adad-ccbe3032d13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.811583 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811567    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.811664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.811578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01c97596-ed76-4f32-adad-ccbe3032d13d" (UID: "01c97596-ed76-4f32-adad-ccbe3032d13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.813329 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.813309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c97596-ed76-4f32-adad-ccbe3032d13d-kube-api-access-njld2" (OuterVolumeSpecName: "kube-api-access-njld2") pod "01c97596-ed76-4f32-adad-ccbe3032d13d" (UID: "01c97596-ed76-4f32-adad-ccbe3032d13d"). InnerVolumeSpecName "kube-api-access-njld2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:48.813599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.813580    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-dshm" (OuterVolumeSpecName: "dshm") pod "01c97596-ed76-4f32-adad-ccbe3032d13d" (UID: "01c97596-ed76-4f32-adad-ccbe3032d13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.813651 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.813601    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c97596-ed76-4f32-adad-ccbe3032d13d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01c97596-ed76-4f32-adad-ccbe3032d13d" (UID: "01c97596-ed76-4f32-adad-ccbe3032d13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:48.912535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.912502    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c97596-ed76-4f32-adad-ccbe3032d13d-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.912535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.912529    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.912535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.912541    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.912748 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.912554    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c97596-ed76-4f32-adad-ccbe3032d13d-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.912748 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:48.912565    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njld2\" (UniqueName: \"kubernetes.io/projected/01c97596-ed76-4f32-adad-ccbe3032d13d-kube-api-access-njld2\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:49.015924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.015884    2578 generic.go:356] "Generic (PLEG): container finished" podID="01c97596-ed76-4f32-adad-ccbe3032d13d" containerID="6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79" exitCode=0
Sep 07 11:38:49.015924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.015919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"
Sep 07 11:38:49.016175 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.015934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx" event={"ID":"01c97596-ed76-4f32-adad-ccbe3032d13d","Type":"ContainerDied","Data":"6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79"}
Sep 07 11:38:49.016175 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.015964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx" event={"ID":"01c97596-ed76-4f32-adad-ccbe3032d13d","Type":"ContainerDied","Data":"4c00f44bf1c6355b2080165dd3a6e47c09a762378448d60698db2338335ffe81"}
Sep 07 11:38:49.016175 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.015980    2578 scope.go:117] "RemoveContainer" containerID="6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79"
Sep 07 11:38:49.028476 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.028445    2578 scope.go:117] "RemoveContainer" containerID="6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79"
Sep 07 11:38:49.028750 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:38:49.028729    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79\": container with ID starting with 6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79 not found: ID does not exist" containerID="6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79"
Sep 07 11:38:49.028826 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.028762    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79"} err="failed to get container status \"6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79\": rpc error: code = NotFound desc = could not find container \"6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79\": container with ID starting with 6c6a32c63f969ee99dae185b654124089c38e631d9be386af96e064dbed3ce79 not found: ID does not exist"
Sep 07 11:38:49.035321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.035299    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"]
Sep 07 11:38:49.041016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:49.040996    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-152c50/autores-model-152c50-kserve-58bf8b5899-zhmtx"]
Sep 07 11:38:50.960574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:50.960540    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c97596-ed76-4f32-adad-ccbe3032d13d" path="/var/lib/kubelet/pods/01c97596-ed76-4f32-adad-ccbe3032d13d/volumes"
Sep 07 11:38:51.033122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:51.033084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:51.130167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:51.130136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:51.253854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:51.253774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"]
Sep 07 11:38:51.254028 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:51.253996    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy" containerID="cri-o://353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9" gracePeriod=30
Sep 07 11:38:51.274697 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:51.274641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:51.754441 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:51.754406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:52.072270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:52.072190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:52.689911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:52.689863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:52.790152 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:52.790119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:53.312724 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.312695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:53.352717 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.352681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"]
Sep 07 11:38:53.352960 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.352929    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn" podUID="4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" containerName="payload-pre-processing" containerID="cri-o://20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319" gracePeriod=30
Sep 07 11:38:53.432349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.432317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:53.532110 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.532081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:53.606639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.606618    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:53.633631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.633600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:53.713193 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.713160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:53.757446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.757416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4f6x\" (UniqueName: \"kubernetes.io/projected/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-kube-api-access-w4f6x\") pod \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") "
Sep 07 11:38:53.757620 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.757489    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-plugins-config-volume\") pod \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\" (UID: \"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1\") "
Sep 07 11:38:53.757927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.757896    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" (UID: "4ee5eeb7-a486-4c68-8d8b-e8457aa580a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:53.759508 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.759476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-kube-api-access-w4f6x" (OuterVolumeSpecName: "kube-api-access-w4f6x") pod "4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" (UID: "4ee5eeb7-a486-4c68-8d8b-e8457aa580a1"). InnerVolumeSpecName "kube-api-access-w4f6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:53.858269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.858195    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4f6x\" (UniqueName: \"kubernetes.io/projected/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-kube-api-access-w4f6x\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:53.858269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:53.858222    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:54.048775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.048740    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" containerID="20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319" exitCode=0
Sep 07 11:38:54.048941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.048780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"
Sep 07 11:38:54.048941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.048823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn" event={"ID":"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1","Type":"ContainerDied","Data":"20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319"}
Sep 07 11:38:54.048941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.048872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn" event={"ID":"4ee5eeb7-a486-4c68-8d8b-e8457aa580a1","Type":"ContainerDied","Data":"d99cb751cd45ca54e167429d400f23585d227adf63a8640d565386610d27d621"}
Sep 07 11:38:54.048941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.048893    2578 scope.go:117] "RemoveContainer" containerID="20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319"
Sep 07 11:38:54.058664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.058648    2578 scope.go:117] "RemoveContainer" containerID="20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319"
Sep 07 11:38:54.058983 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:38:54.058961    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319\": container with ID starting with 20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319 not found: ID does not exist" containerID="20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319"
Sep 07 11:38:54.059048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.058995    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319"} err="failed to get container status \"20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319\": rpc error: code = NotFound desc = could not find container \"20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319\": container with ID starting with 20453529e3eb83dd3a7c156222596dbaba505b737cd47a5def091304d8bd7319 not found: ID does not exist"
Sep 07 11:38:54.071275 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.071249    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"]
Sep 07 11:38:54.079412 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.079391    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f8729d3-6978b6d6b6-rvmsn"]
Sep 07 11:38:54.420009 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.419978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:54.520584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.520552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:54.622254 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.622224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:54.959855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:54.959813    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" path="/var/lib/kubelet/pods/4ee5eeb7-a486-4c68-8d8b-e8457aa580a1/volumes"
Sep 07 11:38:55.132060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:55.132026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:55.227822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:55.227737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:56.041388 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.041346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:56.269415 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.269389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:56.332287 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.332199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:56.458647 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.458613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:56.515623 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.515601    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:38:56.684873 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.684842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-certs\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.684884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6bhc\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-kube-api-access-q6bhc\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.684929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-data\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.684950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-envoy\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.684978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb07e9-6f5e-406c-9072-a58221b680c6-istiod-ca-cert\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.684993    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-credential-socket\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685041 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-podinfo\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685366 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-token\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685366 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-socket\") pod \"96cb07e9-6f5e-406c-9072-a58221b680c6\" (UID: \"96cb07e9-6f5e-406c-9072-a58221b680c6\") "
Sep 07 11:38:56.685366 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:56.685366 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685348    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:56.685599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:56.685599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cb07e9-6f5e-406c-9072-a58221b680c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:56.685599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.685495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:56.687227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.687202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:38:56.687332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.687268    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:56.687332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.687315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-kube-api-access-q6bhc" (OuterVolumeSpecName: "kube-api-access-q6bhc") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "kube-api-access-q6bhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:56.687438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.687378    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "96cb07e9-6f5e-406c-9072-a58221b680c6" (UID: "96cb07e9-6f5e-406c-9072-a58221b680c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:56.754982 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.754951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:56.786789 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786763    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786789 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786788    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786801    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb07e9-6f5e-406c-9072-a58221b680c6-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786809    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786819    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786828    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786835    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786843    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb07e9-6f5e-406c-9072-a58221b680c6-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.786966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.786851    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bhc\" (UniqueName: \"kubernetes.io/projected/96cb07e9-6f5e-406c-9072-a58221b680c6-kube-api-access-q6bhc\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:38:56.832314 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.832273    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:38:56.862011 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.861981    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"]
Sep 07 11:38:56.862376 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862364    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6c951a-0e38-475a-9959-394665f287b2" containerName="payload-pre-processing"
Sep 07 11:38:56.862420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862378    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6c951a-0e38-475a-9959-394665f287b2" containerName="payload-pre-processing"
Sep 07 11:38:56.862420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862387    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c97596-ed76-4f32-adad-ccbe3032d13d" containerName="main"
Sep 07 11:38:56.862420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862394    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c97596-ed76-4f32-adad-ccbe3032d13d" containerName="main"
Sep 07 11:38:56.862420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862402    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77dd24b-8110-4a06-8bab-95ef6f84e1ea" containerName="payload-processing"
Sep 07 11:38:56.862420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862407    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77dd24b-8110-4a06-8bab-95ef6f84e1ea" containerName="payload-processing"
Sep 07 11:38:56.862420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862419    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862425    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862439    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862444    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" containerName="payload-pre-processing"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862449    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" containerName="payload-pre-processing"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862459    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423cd5d8-017d-4d8b-83a2-b189a7b9c9ed" containerName="revoke-keys"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862478    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="423cd5d8-017d-4d8b-83a2-b189a7b9c9ed" containerName="revoke-keys"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6c951a-0e38-475a-9959-394665f287b2" containerName="payload-pre-processing"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862547    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862552    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77dd24b-8110-4a06-8bab-95ef6f84e1ea" containerName="payload-processing"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862558    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="423cd5d8-017d-4d8b-83a2-b189a7b9c9ed" containerName="revoke-keys"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862564    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee5eeb7-a486-4c68-8d8b-e8457aa580a1" containerName="payload-pre-processing"
Sep 07 11:38:56.862643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.862570    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c97596-ed76-4f32-adad-ccbe3032d13d" containerName="main"
Sep 07 11:38:56.865881 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.865865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:56.868381 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.868358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-bf198c-dockercfg-x25th\""
Sep 07 11:38:56.868705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.868687    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-bf198c\""
Sep 07 11:38:56.874323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.874298    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"]
Sep 07 11:38:56.892018 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.891983    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"]
Sep 07 11:38:56.892690 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.892668    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerName="istio-proxy"
Sep 07 11:38:56.896131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.896110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:56.903991 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.903968    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"]
Sep 07 11:38:56.988875 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.988846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrlh\" (UniqueName: \"kubernetes.io/projected/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-kube-api-access-2wrlh\") pod \"payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:56.989015 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.988895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f55236-94b3-4c86-90ea-7d08bc67b87b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") " pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:56.989015 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.988987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5h5t\" (UniqueName: \"kubernetes.io/projected/e7f55236-94b3-4c86-90ea-7d08bc67b87b-kube-api-access-w5h5t\") pod \"payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") " pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:56.989092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:56.989021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:57.069056 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.069019    2578 generic.go:356] "Generic (PLEG): container finished" podID="96cb07e9-6f5e-406c-9072-a58221b680c6" containerID="353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9" exitCode=0
Sep 07 11:38:57.069407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.069089    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"
Sep 07 11:38:57.069407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.069103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" event={"ID":"96cb07e9-6f5e-406c-9072-a58221b680c6","Type":"ContainerDied","Data":"353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9"}
Sep 07 11:38:57.069407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.069156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8" event={"ID":"96cb07e9-6f5e-406c-9072-a58221b680c6","Type":"ContainerDied","Data":"5cf667ae20952a2192af33939f9a15a32ab93483ecd88aaa12e9c0efaea18049"}
Sep 07 11:38:57.069407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.069181    2578 scope.go:117] "RemoveContainer" containerID="353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9"
Sep 07 11:38:57.078171 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.078154    2578 scope.go:117] "RemoveContainer" containerID="5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83"
Sep 07 11:38:57.087166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.087145    2578 scope.go:117] "RemoveContainer" containerID="353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9"
Sep 07 11:38:57.087483 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:38:57.087443    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9\": container with ID starting with 353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9 not found: ID does not exist" containerID="353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9"
Sep 07 11:38:57.087543 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.087493    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9"} err="failed to get container status \"353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9\": rpc error: code = NotFound desc = could not find container \"353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9\": container with ID starting with 353e0540e27a311017c25a6d7c10d69c912f0f1784ce9b03a065f3b012238aa9 not found: ID does not exist"
Sep 07 11:38:57.087543 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.087512    2578 scope.go:117] "RemoveContainer" containerID="5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83"
Sep 07 11:38:57.087816 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:38:57.087796    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83\": container with ID starting with 5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83 not found: ID does not exist" containerID="5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83"
Sep 07 11:38:57.087862 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.087822    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83"} err="failed to get container status \"5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83\": rpc error: code = NotFound desc = could not find container \"5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83\": container with ID starting with 5b7f2cafa11020b0c6f0a6ee884d82307b48e1683f3382a785e6a57cd7568f83 not found: ID does not exist"
Sep 07 11:38:57.089755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.089727    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"]
Sep 07 11:38:57.090073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.090051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f55236-94b3-4c86-90ea-7d08bc67b87b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") " pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:57.090124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.090107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5h5t\" (UniqueName: \"kubernetes.io/projected/e7f55236-94b3-4c86-90ea-7d08bc67b87b-kube-api-access-w5h5t\") pod \"payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") " pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:57.090163 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.090126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:57.090253 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.090237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrlh\" (UniqueName: \"kubernetes.io/projected/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-kube-api-access-2wrlh\") pod \"payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:57.091053 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.091019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f55236-94b3-4c86-90ea-7d08bc67b87b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") " pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:57.091275 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.091253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:57.093645 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.093625    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-152c50-openshift-default-8479f4444b-mgbt8"]
Sep 07 11:38:57.099609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.099584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5h5t\" (UniqueName: \"kubernetes.io/projected/e7f55236-94b3-4c86-90ea-7d08bc67b87b-kube-api-access-w5h5t\") pod \"payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") " pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:57.099696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.099635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrlh\" (UniqueName: \"kubernetes.io/projected/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-kube-api-access-2wrlh\") pod \"payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:57.178730 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.178707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:57.207272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.207248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:57.415184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.415143    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:38:57.451570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.451547    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"]
Sep 07 11:38:57.453604 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:38:57.453584    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9c52db_b8db_4c1a_9049_5a804fd6c72b.slice/crio-8e0ae37186bb9268fe423d4c0775781c8b1c2b40a811fd9bc8106ee806157839 WatchSource:0}: Error finding container 8e0ae37186bb9268fe423d4c0775781c8b1c2b40a811fd9bc8106ee806157839: Status 404 returned error can't find the container with id 8e0ae37186bb9268fe423d4c0775781c8b1c2b40a811fd9bc8106ee806157839
Sep 07 11:38:57.485323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:57.485301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"]
Sep 07 11:38:57.490596 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:38:57.490572    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f55236_94b3_4c86_90ea_7d08bc67b87b.slice/crio-ff8dcc0b447728444395e86956bb6b151d4709bb6aab3404c9b705f58a58bf23 WatchSource:0}: Error finding container ff8dcc0b447728444395e86956bb6b151d4709bb6aab3404c9b705f58a58bf23: Status 404 returned error can't find the container with id ff8dcc0b447728444395e86956bb6b151d4709bb6aab3404c9b705f58a58bf23
Sep 07 11:38:58.078485 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.078415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd" event={"ID":"3b9c52db-b8db-4c1a-9049-5a804fd6c72b","Type":"ContainerStarted","Data":"ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966"}
Sep 07 11:38:58.078919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.078506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:38:58.078919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.078534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd" event={"ID":"3b9c52db-b8db-4c1a-9049-5a804fd6c72b","Type":"ContainerStarted","Data":"8e0ae37186bb9268fe423d4c0775781c8b1c2b40a811fd9bc8106ee806157839"}
Sep 07 11:38:58.080144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.080120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8" event={"ID":"e7f55236-94b3-4c86-90ea-7d08bc67b87b","Type":"ContainerStarted","Data":"157068d8a7399bc3f0d292d2b6973b3c38253231f5fdd66efb184ca703db8e6e"}
Sep 07 11:38:58.080302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.080165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8" event={"ID":"e7f55236-94b3-4c86-90ea-7d08bc67b87b","Type":"ContainerStarted","Data":"ff8dcc0b447728444395e86956bb6b151d4709bb6aab3404c9b705f58a58bf23"}
Sep 07 11:38:58.080302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.080195    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:38:58.097210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.097161    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd" podStartSLOduration=2.097145764 podStartE2EDuration="2.097145764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:58.094762563 +0000 UTC m=+969.569610712" watchObservedRunningTime="2026-09-07 11:38:58.097145764 +0000 UTC m=+969.571993914"
Sep 07 11:38:58.111151 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.111103    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8" podStartSLOduration=2.11108852 podStartE2EDuration="2.11108852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:58.109228347 +0000 UTC m=+969.584076521" watchObservedRunningTime="2026-09-07 11:38:58.11108852 +0000 UTC m=+969.585936668"
Sep 07 11:38:58.959972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:38:58.959939    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96cb07e9-6f5e-406c-9072-a58221b680c6" path="/var/lib/kubelet/pods/96cb07e9-6f5e-406c-9072-a58221b680c6/volumes"
Sep 07 11:39:02.624321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:02.624279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:02.731874 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:02.731843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:02.824686 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:02.824652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:02.926019 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:02.925932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:03.360459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:03.360361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:09.088216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:09.088185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:39:09.088673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:09.088259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:39:14.097169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:14.097086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:39:16.794129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.794094    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"]
Sep 07 11:39:16.797678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.797661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.801140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.801111    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef61c5\"/\"kube-root-ca.crt\""
Sep 07 11:39:16.801271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.801119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef61c5\"/\"openshift-service-ca.crt\""
Sep 07 11:39:16.801271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.801172    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef61c5\"/\"test-model-ef61c5-kserve-self-signed-certs\""
Sep 07 11:39:16.801271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.801188    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef61c5\"/\"default-dockercfg-mpqgp\""
Sep 07 11:39:16.809563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.809539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"]
Sep 07 11:39:16.831360 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.831335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:16.853220 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.853197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-model-cache\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.853326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.853222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdlz\" (UniqueName: \"kubernetes.io/projected/3eba811e-35d4-4d7c-9c57-96ef89c26c26-kube-api-access-jfdlz\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.853326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.853245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tmp-dir\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.853407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.853339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tls-certs\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.853449 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.853406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-home\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.853449 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.853421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-dshm\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.954630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.954590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-home\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.954809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.954635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-dshm\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.954809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.954683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-model-cache\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.954809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.954710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdlz\" (UniqueName: \"kubernetes.io/projected/3eba811e-35d4-4d7c-9c57-96ef89c26c26-kube-api-access-jfdlz\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.954809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.954740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tmp-dir\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.954809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.954803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tls-certs\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.955068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.955006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-home\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.955132 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.955093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-model-cache\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.955262 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.955233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tmp-dir\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.956910 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.956885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-dshm\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.957272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.957255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tls-certs\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:16.962388 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:16.962365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdlz\" (UniqueName: \"kubernetes.io/projected/3eba811e-35d4-4d7c-9c57-96ef89c26c26-kube-api-access-jfdlz\") pod \"test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") " pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:17.107559 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:17.107479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:17.388667 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:17.388633    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"]
Sep 07 11:39:17.389997 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:39:17.389970    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eba811e_35d4_4d7c_9c57_96ef89c26c26.slice/crio-7d66bad21327671c9551a68241c4a62a255e228b97da30590b16816d0b6441f5 WatchSource:0}: Error finding container 7d66bad21327671c9551a68241c4a62a255e228b97da30590b16816d0b6441f5: Status 404 returned error can't find the container with id 7d66bad21327671c9551a68241c4a62a255e228b97da30590b16816d0b6441f5
Sep 07 11:39:18.208853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:18.208816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6" event={"ID":"3eba811e-35d4-4d7c-9c57-96ef89c26c26","Type":"ContainerStarted","Data":"921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75"}
Sep 07 11:39:18.208853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:18.208858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6" event={"ID":"3eba811e-35d4-4d7c-9c57-96ef89c26c26","Type":"ContainerStarted","Data":"7d66bad21327671c9551a68241c4a62a255e228b97da30590b16816d0b6441f5"}
Sep 07 11:39:18.231874 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:18.231827    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6" podStartSLOduration=2.231812155 podStartE2EDuration="2.231812155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:39:18.228876685 +0000 UTC m=+989.703724864" watchObservedRunningTime="2026-09-07 11:39:18.231812155 +0000 UTC m=+989.706660303"
Sep 07 11:39:20.027244 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:20.027207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:27.107964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:27.107931    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:27.107964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:27.107973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:27.120400 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:27.120374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:27.270893 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:27.270869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:39:29.323061 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:29.323025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:31.411512 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:31.411476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:39:35.719387 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:35.719355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:35.775879 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:35.775850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:35.965415 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:35.965379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:36.065801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:36.065696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:45.999034 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:45.998997    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:39:48.217775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.217736    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"]
Sep 07 11:39:48.221263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.221246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.225727 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.225705    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bf198c\"/\"explicit-model-bf198c-kserve-self-signed-certs\""
Sep 07 11:39:48.225841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.225770    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bf198c\"/\"kube-root-ca.crt\""
Sep 07 11:39:48.225898 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.225848    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bf198c\"/\"openshift-service-ca.crt\""
Sep 07 11:39:48.225967 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.225952    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bf198c\"/\"default-dockercfg-lj8nw\""
Sep 07 11:39:48.234423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.234400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"]
Sep 07 11:39:48.251347 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.251324    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:48.328712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.328685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-model-cache\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.328828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.328748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-dshm\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.328871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.328826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-home\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.328871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.328863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tmp-dir\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.328938 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.328893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.328938 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.328930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp4qz\" (UniqueName: \"kubernetes.io/projected/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-kube-api-access-wp4qz\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430351 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-dshm\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-home\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tmp-dir\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp4qz\" (UniqueName: \"kubernetes.io/projected/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-kube-api-access-wp4qz\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-model-cache\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430876 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-home\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430876 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tmp-dir\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.430953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.430866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-model-cache\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.432681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.432660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-dshm\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.432838 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.432820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.441139 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.441119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp4qz\" (UniqueName: \"kubernetes.io/projected/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-kube-api-access-wp4qz\") pod \"explicit-model-bf198c-kserve-849bb54795-hds6h\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") " pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.531271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.531213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:48.804606 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:48.804578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"]
Sep 07 11:39:48.806350 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:39:48.806319    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cf5d495_490f_46e2_9b70_39daf0f4b7f7.slice/crio-5c572f6d706198c84a863b92f77aa5b6570dda2bf478965809fc511a35e4d596 WatchSource:0}: Error finding container 5c572f6d706198c84a863b92f77aa5b6570dda2bf478965809fc511a35e4d596: Status 404 returned error can't find the container with id 5c572f6d706198c84a863b92f77aa5b6570dda2bf478965809fc511a35e4d596
Sep 07 11:39:49.385331 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:49.385294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h" event={"ID":"7cf5d495-490f-46e2-9b70-39daf0f4b7f7","Type":"ContainerStarted","Data":"5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46"}
Sep 07 11:39:49.385331 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:49.385335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h" event={"ID":"7cf5d495-490f-46e2-9b70-39daf0f4b7f7","Type":"ContainerStarted","Data":"5c572f6d706198c84a863b92f77aa5b6570dda2bf478965809fc511a35e4d596"}
Sep 07 11:39:52.282776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:52.282730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:58.352303 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:58.352269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:58.454603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:58.454558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:58.531591 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:58.531556    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:58.531768 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:58.531629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:58.544144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:58.544125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:58.557049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:58.557026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:39:58.573071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:58.573029    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h" podStartSLOduration=10.573016152 podStartE2EDuration="10.573016152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:39:49.403785624 +0000 UTC m=+1020.878633771" watchObservedRunningTime="2026-09-07 11:39:58.573016152 +0000 UTC m=+1030.047864301"
Sep 07 11:39:59.455816 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:59.455784    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:39:59.684381 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:59.684344    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"]
Sep 07 11:39:59.716568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:39:59.716502    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:00.642286 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:00.642250    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-self-signed-certs: secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:00.642756 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:00.642339    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs podName:7cf5d495-490f-46e2-9b70-39daf0f4b7f7 nodeName:}" failed. No retries permitted until 2026-09-07 11:40:01.142320551 +0000 UTC m=+1032.617168679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs") pod "explicit-model-bf198c-kserve-849bb54795-hds6h" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7") : secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:01.146760 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:01.146716    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-self-signed-certs: secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:01.146970 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:01.146952    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs podName:7cf5d495-490f-46e2-9b70-39daf0f4b7f7 nodeName:}" failed. No retries permitted until 2026-09-07 11:40:02.146931916 +0000 UTC m=+1033.621780057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs") pod "explicit-model-bf198c-kserve-849bb54795-hds6h" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7") : secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:02.156372 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:02.156329    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-self-signed-certs: secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:02.156887 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:02.156424    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs podName:7cf5d495-490f-46e2-9b70-39daf0f4b7f7 nodeName:}" failed. No retries permitted until 2026-09-07 11:40:04.156403934 +0000 UTC m=+1035.631252066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs") pod "explicit-model-bf198c-kserve-849bb54795-hds6h" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7") : secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:04.172758 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:04.172724    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-self-signed-certs: secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:04.173131 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:04.172789    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs podName:7cf5d495-490f-46e2-9b70-39daf0f4b7f7 nodeName:}" failed. No retries permitted until 2026-09-07 11:40:08.17277658 +0000 UTC m=+1039.647624706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs") pod "explicit-model-bf198c-kserve-849bb54795-hds6h" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7") : secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:08.207253 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:08.207215    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-self-signed-certs: secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:08.207699 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:08.207292    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs podName:7cf5d495-490f-46e2-9b70-39daf0f4b7f7 nodeName:}" failed. No retries permitted until 2026-09-07 11:40:16.207277889 +0000 UTC m=+1047.682126016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs") pod "explicit-model-bf198c-kserve-849bb54795-hds6h" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7") : secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:09.170923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:09.170875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:09.352572 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:09.352532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:09.411084 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:09.411060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:09.996146 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:09.996113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:10.335407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.335306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:10.364913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.364880    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"]
Sep 07 11:40:10.365288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.365144    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8" podUID="e7f55236-94b3-4c86-90ea-7d08bc67b87b" containerName="payload-processing" containerID="cri-o://157068d8a7399bc3f0d292d2b6973b3c38253231f5fdd66efb184ca703db8e6e" gracePeriod=30
Sep 07 11:40:10.378031 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.378006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"]
Sep 07 11:40:10.436962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.436929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:10.514556 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.514526    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7f55236-94b3-4c86-90ea-7d08bc67b87b" containerID="157068d8a7399bc3f0d292d2b6973b3c38253231f5fdd66efb184ca703db8e6e" exitCode=0
Sep 07 11:40:10.514707 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.514594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8" event={"ID":"e7f55236-94b3-4c86-90ea-7d08bc67b87b","Type":"ContainerDied","Data":"157068d8a7399bc3f0d292d2b6973b3c38253231f5fdd66efb184ca703db8e6e"}
Sep 07 11:40:10.514809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.514789    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd" podUID="3b9c52db-b8db-4c1a-9049-5a804fd6c72b" containerName="payload-pre-processing" containerID="cri-o://ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966" gracePeriod=30
Sep 07 11:40:10.534990 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.534958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:10.640752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.640728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:10.641924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.641908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:40:10.730545 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.730519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f55236-94b3-4c86-90ea-7d08bc67b87b-plugins-config-volume\") pod \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") "
Sep 07 11:40:10.730682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.730578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5h5t\" (UniqueName: \"kubernetes.io/projected/e7f55236-94b3-4c86-90ea-7d08bc67b87b-kube-api-access-w5h5t\") pod \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\" (UID: \"e7f55236-94b3-4c86-90ea-7d08bc67b87b\") "
Sep 07 11:40:10.730864 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.730842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f55236-94b3-4c86-90ea-7d08bc67b87b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7f55236-94b3-4c86-90ea-7d08bc67b87b" (UID: "e7f55236-94b3-4c86-90ea-7d08bc67b87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:40:10.732631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.732606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f55236-94b3-4c86-90ea-7d08bc67b87b-kube-api-access-w5h5t" (OuterVolumeSpecName: "kube-api-access-w5h5t") pod "e7f55236-94b3-4c86-90ea-7d08bc67b87b" (UID: "e7f55236-94b3-4c86-90ea-7d08bc67b87b"). InnerVolumeSpecName "kube-api-access-w5h5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:10.766635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.766606    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:40:10.839176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.839054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-plugins-config-volume\") pod \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") "
Sep 07 11:40:10.839442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.839419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wrlh\" (UniqueName: \"kubernetes.io/projected/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-kube-api-access-2wrlh\") pod \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\" (UID: \"3b9c52db-b8db-4c1a-9049-5a804fd6c72b\") "
Sep 07 11:40:10.840016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.839874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b9c52db-b8db-4c1a-9049-5a804fd6c72b" (UID: "3b9c52db-b8db-4c1a-9049-5a804fd6c72b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:40:10.840124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.840016    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:10.840124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.840030    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f55236-94b3-4c86-90ea-7d08bc67b87b-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:10.840124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.840038    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5h5t\" (UniqueName: \"kubernetes.io/projected/e7f55236-94b3-4c86-90ea-7d08bc67b87b-kube-api-access-w5h5t\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:10.841774 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.841733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-kube-api-access-2wrlh" (OuterVolumeSpecName: "kube-api-access-2wrlh") pod "3b9c52db-b8db-4c1a-9049-5a804fd6c72b" (UID: "3b9c52db-b8db-4c1a-9049-5a804fd6c72b"). InnerVolumeSpecName "kube-api-access-2wrlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:10.938066 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.938038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:10.941255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:10.941236    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrlh\" (UniqueName: \"kubernetes.io/projected/3b9c52db-b8db-4c1a-9049-5a804fd6c72b-kube-api-access-2wrlh\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:11.522114 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.522073    2578 generic.go:356] "Generic (PLEG): container finished" podID="3b9c52db-b8db-4c1a-9049-5a804fd6c72b" containerID="ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966" exitCode=0
Sep 07 11:40:11.522595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.522197    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"
Sep 07 11:40:11.522595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.522211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd" event={"ID":"3b9c52db-b8db-4c1a-9049-5a804fd6c72b","Type":"ContainerDied","Data":"ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966"}
Sep 07 11:40:11.522595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.522266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd" event={"ID":"3b9c52db-b8db-4c1a-9049-5a804fd6c72b","Type":"ContainerDied","Data":"8e0ae37186bb9268fe423d4c0775781c8b1c2b40a811fd9bc8106ee806157839"}
Sep 07 11:40:11.522595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.522298    2578 scope.go:117] "RemoveContainer" containerID="ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966"
Sep 07 11:40:11.523948 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.523913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8" event={"ID":"e7f55236-94b3-4c86-90ea-7d08bc67b87b","Type":"ContainerDied","Data":"ff8dcc0b447728444395e86956bb6b151d4709bb6aab3404c9b705f58a58bf23"}
Sep 07 11:40:11.523948 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.523933    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"
Sep 07 11:40:11.533305 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.533288    2578 scope.go:117] "RemoveContainer" containerID="ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966"
Sep 07 11:40:11.533628 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:11.533604    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966\": container with ID starting with ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966 not found: ID does not exist" containerID="ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966"
Sep 07 11:40:11.533688 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.533637    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966"} err="failed to get container status \"ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966\": rpc error: code = NotFound desc = could not find container \"ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966\": container with ID starting with ecd87a6cd7978c46b051c51f4b557cddbb3bd11df968ba2329abc557afe8b966 not found: ID does not exist"
Sep 07 11:40:11.533688 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.533658    2578 scope.go:117] "RemoveContainer" containerID="157068d8a7399bc3f0d292d2b6973b3c38253231f5fdd66efb184ca703db8e6e"
Sep 07 11:40:11.543918 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.543897    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"]
Sep 07 11:40:11.546321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.546297    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bf198c-5d79c744bb-fk6pd"]
Sep 07 11:40:11.555446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.555424    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"]
Sep 07 11:40:11.560996 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:11.560974    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bf198c-7cc556c7fc-mb9w8"]
Sep 07 11:40:12.960668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:12.960635    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9c52db-b8db-4c1a-9049-5a804fd6c72b" path="/var/lib/kubelet/pods/3b9c52db-b8db-4c1a-9049-5a804fd6c72b/volumes"
Sep 07 11:40:12.961047 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:12.961012    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f55236-94b3-4c86-90ea-7d08bc67b87b" path="/var/lib/kubelet/pods/e7f55236-94b3-4c86-90ea-7d08bc67b87b/volumes"
Sep 07 11:40:15.857149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:15.857119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:15.960680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:15.960631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:16.057818 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.057787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:16.158336 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.158308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:16.259278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.259237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:16.288533 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:16.288512    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-self-signed-certs: secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:16.288652 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:16.288573    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs podName:7cf5d495-490f-46e2-9b70-39daf0f4b7f7 nodeName:}" failed. No retries permitted until 2026-09-07 11:40:32.288560556 +0000 UTC m=+1063.763408682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs") pod "explicit-model-bf198c-kserve-849bb54795-hds6h" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7") : secret "explicit-model-bf198c-kserve-self-signed-certs" not found
Sep 07 11:40:16.477960 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.477874    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h" podUID="7cf5d495-490f-46e2-9b70-39daf0f4b7f7" containerName="main" containerID="cri-o://5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46" gracePeriod=45
Sep 07 11:40:16.501140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.501115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:40:16.737986 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.737921    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:40:16.793799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.793769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs\") pod \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") "
Sep 07 11:40:16.793951 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.793808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp4qz\" (UniqueName: \"kubernetes.io/projected/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-kube-api-access-wp4qz\") pod \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") "
Sep 07 11:40:16.793951 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.793848    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-dshm\") pod \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") "
Sep 07 11:40:16.793951 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.793875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-home\") pod \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") "
Sep 07 11:40:16.793951 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.793917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-model-cache\") pod \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") "
Sep 07 11:40:16.794104 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.793965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tmp-dir\") pod \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\" (UID: \"7cf5d495-490f-46e2-9b70-39daf0f4b7f7\") "
Sep 07 11:40:16.794205 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.794173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-home" (OuterVolumeSpecName: "home") pod "7cf5d495-490f-46e2-9b70-39daf0f4b7f7" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:16.794402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.794204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "7cf5d495-490f-46e2-9b70-39daf0f4b7f7" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:16.794402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.794289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7cf5d495-490f-46e2-9b70-39daf0f4b7f7" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:16.796343 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.796317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7cf5d495-490f-46e2-9b70-39daf0f4b7f7" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:16.796713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.796654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-dshm" (OuterVolumeSpecName: "dshm") pod "7cf5d495-490f-46e2-9b70-39daf0f4b7f7" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:16.796821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.796761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-kube-api-access-wp4qz" (OuterVolumeSpecName: "kube-api-access-wp4qz") pod "7cf5d495-490f-46e2-9b70-39daf0f4b7f7" (UID: "7cf5d495-490f-46e2-9b70-39daf0f4b7f7"). InnerVolumeSpecName "kube-api-access-wp4qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:16.821128 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.821100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:16.895371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.895345    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:16.895371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.895373    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp4qz\" (UniqueName: \"kubernetes.io/projected/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-kube-api-access-wp4qz\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:16.895755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.895388    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:16.895755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.895401    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:16.895755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.895411    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:16.895755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.895422    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf5d495-490f-46e2-9b70-39daf0f4b7f7-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:16.928710 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:16.928680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:17.029418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.029347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:17.561234 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.561193    2578 generic.go:356] "Generic (PLEG): container finished" podID="7cf5d495-490f-46e2-9b70-39daf0f4b7f7" containerID="5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46" exitCode=0
Sep 07 11:40:17.561408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.561256    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"
Sep 07 11:40:17.561408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.561282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h" event={"ID":"7cf5d495-490f-46e2-9b70-39daf0f4b7f7","Type":"ContainerDied","Data":"5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46"}
Sep 07 11:40:17.561408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.561334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h" event={"ID":"7cf5d495-490f-46e2-9b70-39daf0f4b7f7","Type":"ContainerDied","Data":"5c572f6d706198c84a863b92f77aa5b6570dda2bf478965809fc511a35e4d596"}
Sep 07 11:40:17.561408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.561353    2578 scope.go:117] "RemoveContainer" containerID="5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46"
Sep 07 11:40:17.570740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.570718    2578 scope.go:117] "RemoveContainer" containerID="5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46"
Sep 07 11:40:17.571032 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:17.571010    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46\": container with ID starting with 5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46 not found: ID does not exist" containerID="5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46"
Sep 07 11:40:17.571091 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.571040    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46"} err="failed to get container status \"5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46\": rpc error: code = NotFound desc = could not find container \"5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46\": container with ID starting with 5ec9b26364ab92e1d1246ebba906e055a7601f6ef05fcdc384a2ed38dfb25e46 not found: ID does not exist"
Sep 07 11:40:17.583662 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.583640    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"]
Sep 07 11:40:17.590584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:17.590564    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-bf198c/explicit-model-bf198c-kserve-849bb54795-hds6h"]
Sep 07 11:40:18.571375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:18.571339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:18.959989 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:18.959959    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cf5d495-490f-46e2-9b70-39daf0f4b7f7" path="/var/lib/kubelet/pods/7cf5d495-490f-46e2-9b70-39daf0f4b7f7/volumes"
Sep 07 11:40:22.286081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:22.286050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:22.532973 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:22.532946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:22.969386 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:22.969355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:23.300959 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:23.300876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:23.645263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:23.645235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:24.560820 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:24.560784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:24.626225 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:24.626188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:24.863642 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:24.863613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:24.958544 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:24.958511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:25.058285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:25.058254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:25.255250 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:25.255164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:31.437158 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.437120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:31.519193 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.519161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:31.678315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.678285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:31.979141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979105    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"]
Sep 07 11:40:31.979581 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979560    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f55236-94b3-4c86-90ea-7d08bc67b87b" containerName="payload-processing"
Sep 07 11:40:31.979679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979583    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f55236-94b3-4c86-90ea-7d08bc67b87b" containerName="payload-processing"
Sep 07 11:40:31.979679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979598    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9c52db-b8db-4c1a-9049-5a804fd6c72b" containerName="payload-pre-processing"
Sep 07 11:40:31.979679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979606    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9c52db-b8db-4c1a-9049-5a804fd6c72b" containerName="payload-pre-processing"
Sep 07 11:40:31.979679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979647    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cf5d495-490f-46e2-9b70-39daf0f4b7f7" containerName="main"
Sep 07 11:40:31.979679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979658    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cf5d495-490f-46e2-9b70-39daf0f4b7f7" containerName="main"
Sep 07 11:40:31.979949 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979754    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f55236-94b3-4c86-90ea-7d08bc67b87b" containerName="payload-processing"
Sep 07 11:40:31.979949 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979769    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cf5d495-490f-46e2-9b70-39daf0f4b7f7" containerName="main"
Sep 07 11:40:31.979949 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.979781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9c52db-b8db-4c1a-9049-5a804fd6c72b" containerName="payload-pre-processing"
Sep 07 11:40:31.984047 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.984030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:31.987299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.987272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d6b326a7-openshift-default-dockercfg-ht5l8\""
Sep 07 11:40:31.998381 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:31.998359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"]
Sep 07 11:40:32.013402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.013381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:32.114113 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.114083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:32.122261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwk4l\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-kube-api-access-dwk4l\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122355 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-certs\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122355 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-credential-socket\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-envoy\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-token\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-podinfo\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istiod-ca-cert\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-data\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.122582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.122518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-socket\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-token\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-podinfo\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istiod-ca-cert\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-data\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-socket\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwk4l\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-kube-api-access-dwk4l\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-certs\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-credential-socket\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-envoy\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.223923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-data\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.224016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.223965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-socket\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.224074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.224020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-credential-socket\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.224149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.224130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istiod-ca-cert\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.224204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.224156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-certs\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.225951 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.225925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-envoy\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.226047 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.226017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-podinfo\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.231869 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.231814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwk4l\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-kube-api-access-dwk4l\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.232182 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.232165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-token\") pod \"e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") " pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.295945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.295920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:32.576581 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.576552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"]
Sep 07 11:40:32.577611 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:40:32.577582    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98d72b2_96ea_4d62_a288_b18c8d7db8bd.slice/crio-07ba1b4341fbcb378623fe1a5fa021febbe7389e3d18be882fa5c3b14dbe7c79 WatchSource:0}: Error finding container 07ba1b4341fbcb378623fe1a5fa021febbe7389e3d18be882fa5c3b14dbe7c79: Status 404 returned error can't find the container with id 07ba1b4341fbcb378623fe1a5fa021febbe7389e3d18be882fa5c3b14dbe7c79
Sep 07 11:40:32.579824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.579792    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:40:32.579898 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.579857    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:40:32.579898 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.579885    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:40:32.649901 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.649870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" event={"ID":"d98d72b2-96ea-4d62-a288-b18c8d7db8bd","Type":"ContainerStarted","Data":"29de278882a25632bd1a38038fbfd2df6be13ce797ddead49abf222ea22be7bd"}
Sep 07 11:40:32.650001 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.649928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" event={"ID":"d98d72b2-96ea-4d62-a288-b18c8d7db8bd","Type":"ContainerStarted","Data":"07ba1b4341fbcb378623fe1a5fa021febbe7389e3d18be882fa5c3b14dbe7c79"}
Sep 07 11:40:32.671634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:32.671585    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podStartSLOduration=1.671562826 podStartE2EDuration="1.671562826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:40:32.670910054 +0000 UTC m=+1064.145758227" watchObservedRunningTime="2026-09-07 11:40:32.671562826 +0000 UTC m=+1064.146411003"
Sep 07 11:40:33.297083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:33.297044    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:33.302458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:33.302432    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:33.656369 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:33.656331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:33.657187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:33.657169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:40:34.392691 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.392657    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"]
Sep 07 11:40:34.396273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.396254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.399932 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.399909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f5267c\"/\"noten-model-f5267c-kserve-self-signed-certs\""
Sep 07 11:40:34.400055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.399958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f5267c\"/\"openshift-service-ca.crt\""
Sep 07 11:40:34.400055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.399909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f5267c\"/\"default-dockercfg-vq2qk\""
Sep 07 11:40:34.400055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.399911    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f5267c\"/\"kube-root-ca.crt\""
Sep 07 11:40:34.404993 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.404974    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"]
Sep 07 11:40:34.418451 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.418429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:34.545304 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.545267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-tmp-dir\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.545433 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.545322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-model-cache\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.545433 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.545399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-dshm\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.545551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.545439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvp7\" (UniqueName: \"kubernetes.io/projected/564cf7c7-3582-413b-a204-d62bdd555b84-kube-api-access-smvp7\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.545551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.545487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-home\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.545551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.545521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564cf7c7-3582-413b-a204-d62bdd555b84-tls-certs\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.645969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.645895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564cf7c7-3582-413b-a204-d62bdd555b84-tls-certs\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.645969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.645950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-tmp-dir\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.646149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.645987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-model-cache\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.646149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.646012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-dshm\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.646149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.646039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvp7\" (UniqueName: \"kubernetes.io/projected/564cf7c7-3582-413b-a204-d62bdd555b84-kube-api-access-smvp7\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.646149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.646063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-home\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.646606 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.646584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-model-cache\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.646773 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.646619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-tmp-dir\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.646773 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.646646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-home\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.648953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.648890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564cf7c7-3582-413b-a204-d62bdd555b84-tls-certs\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.648953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.648907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-dshm\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.654308 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.654281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvp7\" (UniqueName: \"kubernetes.io/projected/564cf7c7-3582-413b-a204-d62bdd555b84-kube-api-access-smvp7\") pod \"noten-model-f5267c-kserve-7c594b5f79-tgtz7\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") " pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.707433 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.707397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:34.715405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.715377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:34.807734 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:34.807707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:35.007098 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:35.007072    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"]
Sep 07 11:40:35.008178 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:40:35.008151    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod564cf7c7_3582_413b_a204_d62bdd555b84.slice/crio-9bceafb0b595112ee89f77f9f22f81a2fa40da7478a693c8fd52e69d4d7da4e5 WatchSource:0}: Error finding container 9bceafb0b595112ee89f77f9f22f81a2fa40da7478a693c8fd52e69d4d7da4e5: Status 404 returned error can't find the container with id 9bceafb0b595112ee89f77f9f22f81a2fa40da7478a693c8fd52e69d4d7da4e5
Sep 07 11:40:35.672660 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:35.672624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7" event={"ID":"564cf7c7-3582-413b-a204-d62bdd555b84","Type":"ContainerStarted","Data":"c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5"}
Sep 07 11:40:35.672810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:35.672671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7" event={"ID":"564cf7c7-3582-413b-a204-d62bdd555b84","Type":"ContainerStarted","Data":"9bceafb0b595112ee89f77f9f22f81a2fa40da7478a693c8fd52e69d4d7da4e5"}
Sep 07 11:40:35.694860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:35.694814    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7" podStartSLOduration=1.694799626 podStartE2EDuration="1.694799626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:40:35.692755744 +0000 UTC m=+1067.167603884" watchObservedRunningTime="2026-09-07 11:40:35.694799626 +0000 UTC m=+1067.169647780"
Sep 07 11:40:35.870370 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:35.870335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:35.937605 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:35.937530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:36.040702 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:36.040673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:36.359930 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:36.359846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:36.439387 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:36.439355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:38.208784 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:38.208742    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:40:39.178277 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:39.178236    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"]
Sep 07 11:40:39.204659 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:39.204629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:40.873249 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:40.873215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"]
Sep 07 11:40:40.891925 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:40.891890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:41.016203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.016164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:41.253159 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.253071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:41.374523 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.374491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:41.790602 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.790528    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"]
Sep 07 11:40:41.800416 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.800387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:41.923378 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.920506    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"]
Sep 07 11:40:41.925733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.925707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"
Sep 07 11:40:41.928523 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.928496    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:40:41.928825 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.928803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:40:41.930536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:41.930512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"]
Sep 07 11:40:42.017098 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:42.017066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7442b622-342b-4670-af53-939774b2a0f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf\" (UID: \"7442b622-342b-4670-af53-939774b2a0f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"
Sep 07 11:40:42.118368 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:42.118335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7442b622-342b-4670-af53-939774b2a0f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf\" (UID: \"7442b622-342b-4670-af53-939774b2a0f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"
Sep 07 11:40:42.119003 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:42.118980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7442b622-342b-4670-af53-939774b2a0f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf\" (UID: \"7442b622-342b-4670-af53-939774b2a0f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"
Sep 07 11:40:42.129003 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:42.128973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:42.240204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:42.240175    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"
Sep 07 11:40:42.514586 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:42.514560    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"]
Sep 07 11:40:42.516891 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:40:42.516866    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7442b622_342b_4670_af53_939774b2a0f9.slice/crio-8d1f6d178a1ad313ffc678a094a826e5ee1a64caa1c028a653426a2c00a0cb34 WatchSource:0}: Error finding container 8d1f6d178a1ad313ffc678a094a826e5ee1a64caa1c028a653426a2c00a0cb34: Status 404 returned error can't find the container with id 8d1f6d178a1ad313ffc678a094a826e5ee1a64caa1c028a653426a2c00a0cb34
Sep 07 11:40:42.713358 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:42.713328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf" event={"ID":"7442b622-342b-4670-af53-939774b2a0f9","Type":"ContainerStarted","Data":"8d1f6d178a1ad313ffc678a094a826e5ee1a64caa1c028a653426a2c00a0cb34"}
Sep 07 11:40:43.720517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:43.720485    2578 generic.go:356] "Generic (PLEG): container finished" podID="7442b622-342b-4670-af53-939774b2a0f9" containerID="010199b2009cb22bed8bba31c917dc28806445a2f10d1c2a8867951d3091b2c4" exitCode=0
Sep 07 11:40:43.720888 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:43.720543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf" event={"ID":"7442b622-342b-4670-af53-939774b2a0f9","Type":"ContainerDied","Data":"010199b2009cb22bed8bba31c917dc28806445a2f10d1c2a8867951d3091b2c4"}
Sep 07 11:40:44.707717 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:44.707684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:44.864417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:44.864394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"
Sep 07 11:40:45.045408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:45.045340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7442b622-342b-4670-af53-939774b2a0f9-maas-api-service-ca\") pod \"7442b622-342b-4670-af53-939774b2a0f9\" (UID: \"7442b622-342b-4670-af53-939774b2a0f9\") "
Sep 07 11:40:45.045707 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:45.045685    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7442b622-342b-4670-af53-939774b2a0f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7442b622-342b-4670-af53-939774b2a0f9" (UID: "7442b622-342b-4670-af53-939774b2a0f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:40:45.146133 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:45.146104    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7442b622-342b-4670-af53-939774b2a0f9-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:45.733057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:45.733025    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"
Sep 07 11:40:45.733057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:45.733051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf" event={"ID":"7442b622-342b-4670-af53-939774b2a0f9","Type":"ContainerDied","Data":"8d1f6d178a1ad313ffc678a094a826e5ee1a64caa1c028a653426a2c00a0cb34"}
Sep 07 11:40:45.733275 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:45.733096    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d1f6d178a1ad313ffc678a094a826e5ee1a64caa1c028a653426a2c00a0cb34"
Sep 07 11:40:46.453556 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:46.453516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:50.298236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:50.298194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:40:52.401817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.401780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:52.477212 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.477181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:52.533139 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.533104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:52.592629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.592589    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"]
Sep 07 11:40:52.595083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.595056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:52.595199 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.595063    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b7e4c1-6crbf"]
Sep 07 11:40:52.960011 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.959975    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7442b622-342b-4670-af53-939774b2a0f9" path="/var/lib/kubelet/pods/7442b622-342b-4670-af53-939774b2a0f9/volumes"
Sep 07 11:40:52.973851 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:52.973825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:54.198194 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.198148    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz" podUID="14d69976-10f6-47db-9cae-10327562842f" containerName="main" containerID="cri-o://bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b" gracePeriod=45
Sep 07 11:40:54.449687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.449632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:40:54.525238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d69976-10f6-47db-9cae-10327562842f-tls-certs\") pod \"14d69976-10f6-47db-9cae-10327562842f\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") "
Sep 07 11:40:54.525410 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-model-cache\") pod \"14d69976-10f6-47db-9cae-10327562842f\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") "
Sep 07 11:40:54.525410 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525381    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmhn7\" (UniqueName: \"kubernetes.io/projected/14d69976-10f6-47db-9cae-10327562842f-kube-api-access-tmhn7\") pod \"14d69976-10f6-47db-9cae-10327562842f\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") "
Sep 07 11:40:54.525576 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-home\") pod \"14d69976-10f6-47db-9cae-10327562842f\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") "
Sep 07 11:40:54.525576 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-tmp-dir\") pod \"14d69976-10f6-47db-9cae-10327562842f\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") "
Sep 07 11:40:54.525576 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525498    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-dshm\") pod \"14d69976-10f6-47db-9cae-10327562842f\" (UID: \"14d69976-10f6-47db-9cae-10327562842f\") "
Sep 07 11:40:54.525728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-model-cache" (OuterVolumeSpecName: "model-cache") pod "14d69976-10f6-47db-9cae-10327562842f" (UID: "14d69976-10f6-47db-9cae-10327562842f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:54.525728 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525656    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-home" (OuterVolumeSpecName: "home") pod "14d69976-10f6-47db-9cae-10327562842f" (UID: "14d69976-10f6-47db-9cae-10327562842f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:54.525841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525753    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14d69976-10f6-47db-9cae-10327562842f" (UID: "14d69976-10f6-47db-9cae-10327562842f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:54.525912 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525891    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:54.525956 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525922    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:54.525956 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.525952    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:54.527630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.527608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d69976-10f6-47db-9cae-10327562842f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14d69976-10f6-47db-9cae-10327562842f" (UID: "14d69976-10f6-47db-9cae-10327562842f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:54.527714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.527650    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d69976-10f6-47db-9cae-10327562842f-kube-api-access-tmhn7" (OuterVolumeSpecName: "kube-api-access-tmhn7") pod "14d69976-10f6-47db-9cae-10327562842f" (UID: "14d69976-10f6-47db-9cae-10327562842f"). InnerVolumeSpecName "kube-api-access-tmhn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:54.527714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.527649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-dshm" (OuterVolumeSpecName: "dshm") pod "14d69976-10f6-47db-9cae-10327562842f" (UID: "14d69976-10f6-47db-9cae-10327562842f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:54.627049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.627016    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmhn7\" (UniqueName: \"kubernetes.io/projected/14d69976-10f6-47db-9cae-10327562842f-kube-api-access-tmhn7\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:54.627049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.627044    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d69976-10f6-47db-9cae-10327562842f-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:54.627049 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.627054    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d69976-10f6-47db-9cae-10327562842f-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:54.785573 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.785490    2578 generic.go:356] "Generic (PLEG): container finished" podID="14d69976-10f6-47db-9cae-10327562842f" containerID="bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b" exitCode=0
Sep 07 11:40:54.785573 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.785528    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"
Sep 07 11:40:54.785573 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.785551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz" event={"ID":"14d69976-10f6-47db-9cae-10327562842f","Type":"ContainerDied","Data":"bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b"}
Sep 07 11:40:54.785816 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.785603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz" event={"ID":"14d69976-10f6-47db-9cae-10327562842f","Type":"ContainerDied","Data":"87a05a6ddb0038e05dbba1842d8dff83b9fe7ac8a3dfaf83b8a4f8f46a918494"}
Sep 07 11:40:54.785816 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.785625    2578 scope.go:117] "RemoveContainer" containerID="bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b"
Sep 07 11:40:54.795379 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.795361    2578 scope.go:117] "RemoveContainer" containerID="bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b"
Sep 07 11:40:54.795697 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:54.795681    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b\": container with ID starting with bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b not found: ID does not exist" containerID="bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b"
Sep 07 11:40:54.795744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.795707    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b"} err="failed to get container status \"bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b\": rpc error: code = NotFound desc = could not find container \"bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b\": container with ID starting with bc8ae8ad7f9a0e0a1f601ac0cf1245b3aeb6db184597c79dcebe961e0e98cf8b not found: ID does not exist"
Sep 07 11:40:54.807850 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.807828    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"]
Sep 07 11:40:54.811412 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.811391    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b7e4c1/test-model-b7e4c1-kserve-f8c8dc869-dcljz"]
Sep 07 11:40:54.960165 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:54.960135    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d69976-10f6-47db-9cae-10327562842f" path="/var/lib/kubelet/pods/14d69976-10f6-47db-9cae-10327562842f/volumes"
Sep 07 11:40:55.893551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:55.893494    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7" podUID="564cf7c7-3582-413b-a204-d62bdd555b84" containerName="main" containerID="cri-o://c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5" gracePeriod=45
Sep 07 11:40:56.144513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.144440    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:56.241909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.241882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564cf7c7-3582-413b-a204-d62bdd555b84-tls-certs\") pod \"564cf7c7-3582-413b-a204-d62bdd555b84\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") "
Sep 07 11:40:56.242039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.241928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-dshm\") pod \"564cf7c7-3582-413b-a204-d62bdd555b84\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") "
Sep 07 11:40:56.242039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242006    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smvp7\" (UniqueName: \"kubernetes.io/projected/564cf7c7-3582-413b-a204-d62bdd555b84-kube-api-access-smvp7\") pod \"564cf7c7-3582-413b-a204-d62bdd555b84\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") "
Sep 07 11:40:56.242116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-model-cache\") pod \"564cf7c7-3582-413b-a204-d62bdd555b84\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") "
Sep 07 11:40:56.242116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242105    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-home\") pod \"564cf7c7-3582-413b-a204-d62bdd555b84\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") "
Sep 07 11:40:56.242191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-tmp-dir\") pod \"564cf7c7-3582-413b-a204-d62bdd555b84\" (UID: \"564cf7c7-3582-413b-a204-d62bdd555b84\") "
Sep 07 11:40:56.242377 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-model-cache" (OuterVolumeSpecName: "model-cache") pod "564cf7c7-3582-413b-a204-d62bdd555b84" (UID: "564cf7c7-3582-413b-a204-d62bdd555b84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:56.242377 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-home" (OuterVolumeSpecName: "home") pod "564cf7c7-3582-413b-a204-d62bdd555b84" (UID: "564cf7c7-3582-413b-a204-d62bdd555b84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:56.242598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242454    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:56.242598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242504    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:56.242598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.242512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "564cf7c7-3582-413b-a204-d62bdd555b84" (UID: "564cf7c7-3582-413b-a204-d62bdd555b84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:56.244144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.244121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-dshm" (OuterVolumeSpecName: "dshm") pod "564cf7c7-3582-413b-a204-d62bdd555b84" (UID: "564cf7c7-3582-413b-a204-d62bdd555b84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:56.244208 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.244191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564cf7c7-3582-413b-a204-d62bdd555b84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "564cf7c7-3582-413b-a204-d62bdd555b84" (UID: "564cf7c7-3582-413b-a204-d62bdd555b84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:56.244249 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.244237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564cf7c7-3582-413b-a204-d62bdd555b84-kube-api-access-smvp7" (OuterVolumeSpecName: "kube-api-access-smvp7") pod "564cf7c7-3582-413b-a204-d62bdd555b84" (UID: "564cf7c7-3582-413b-a204-d62bdd555b84"). InnerVolumeSpecName "kube-api-access-smvp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:56.343438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.343415    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:56.343438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.343435    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smvp7\" (UniqueName: \"kubernetes.io/projected/564cf7c7-3582-413b-a204-d62bdd555b84-kube-api-access-smvp7\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:56.343588 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.343446    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564cf7c7-3582-413b-a204-d62bdd555b84-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:56.343588 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.343456    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564cf7c7-3582-413b-a204-d62bdd555b84-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:56.744968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.744932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:56.799398 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.799362    2578 generic.go:356] "Generic (PLEG): container finished" podID="564cf7c7-3582-413b-a204-d62bdd555b84" containerID="c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5" exitCode=0
Sep 07 11:40:56.799549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.799414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7" event={"ID":"564cf7c7-3582-413b-a204-d62bdd555b84","Type":"ContainerDied","Data":"c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5"}
Sep 07 11:40:56.799549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.799446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7" event={"ID":"564cf7c7-3582-413b-a204-d62bdd555b84","Type":"ContainerDied","Data":"9bceafb0b595112ee89f77f9f22f81a2fa40da7478a693c8fd52e69d4d7da4e5"}
Sep 07 11:40:56.799549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.799478    2578 scope.go:117] "RemoveContainer" containerID="c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5"
Sep 07 11:40:56.799549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.799421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"
Sep 07 11:40:56.810565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.810535    2578 scope.go:117] "RemoveContainer" containerID="c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5"
Sep 07 11:40:56.810896 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:56.810871    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5\": container with ID starting with c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5 not found: ID does not exist" containerID="c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5"
Sep 07 11:40:56.811001 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.810901    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5"} err="failed to get container status \"c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5\": rpc error: code = NotFound desc = could not find container \"c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5\": container with ID starting with c11176ce0da904cca41be9fb35e08d5923944c131d5c062f73ed38d2385e43c5 not found: ID does not exist"
Sep 07 11:40:56.811156 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.811129    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6" podUID="3eba811e-35d4-4d7c-9c57-96ef89c26c26" containerName="main" containerID="cri-o://921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75" gracePeriod=45
Sep 07 11:40:56.823070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.823049    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"]
Sep 07 11:40:56.826690 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.826667    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f5267c/noten-model-f5267c-kserve-7c594b5f79-tgtz7"]
Sep 07 11:40:56.960508 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:56.960476    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="564cf7c7-3582-413b-a204-d62bdd555b84" path="/var/lib/kubelet/pods/564cf7c7-3582-413b-a204-d62bdd555b84/volumes"
Sep 07 11:40:57.063040 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.063018    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:40:57.150694 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.150665    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-home\") pod \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") "
Sep 07 11:40:57.150810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.150702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfdlz\" (UniqueName: \"kubernetes.io/projected/3eba811e-35d4-4d7c-9c57-96ef89c26c26-kube-api-access-jfdlz\") pod \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") "
Sep 07 11:40:57.150810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.150756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tls-certs\") pod \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") "
Sep 07 11:40:57.150810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.150776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tmp-dir\") pod \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") "
Sep 07 11:40:57.150978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.150813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-model-cache\") pod \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") "
Sep 07 11:40:57.150978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.150895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-dshm\") pod \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\" (UID: \"3eba811e-35d4-4d7c-9c57-96ef89c26c26\") "
Sep 07 11:40:57.150978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.150960    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-home" (OuterVolumeSpecName: "home") pod "3eba811e-35d4-4d7c-9c57-96ef89c26c26" (UID: "3eba811e-35d4-4d7c-9c57-96ef89c26c26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:57.151112 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.151058    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3eba811e-35d4-4d7c-9c57-96ef89c26c26" (UID: "3eba811e-35d4-4d7c-9c57-96ef89c26c26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:57.151112 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.151066    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-model-cache" (OuterVolumeSpecName: "model-cache") pod "3eba811e-35d4-4d7c-9c57-96ef89c26c26" (UID: "3eba811e-35d4-4d7c-9c57-96ef89c26c26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:57.151238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.151222    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:57.151290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.151240    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:57.151290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.151249    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:57.152961 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.152937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-dshm" (OuterVolumeSpecName: "dshm") pod "3eba811e-35d4-4d7c-9c57-96ef89c26c26" (UID: "3eba811e-35d4-4d7c-9c57-96ef89c26c26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:40:57.153060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.152986    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3eba811e-35d4-4d7c-9c57-96ef89c26c26" (UID: "3eba811e-35d4-4d7c-9c57-96ef89c26c26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:57.153060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.153037    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eba811e-35d4-4d7c-9c57-96ef89c26c26-kube-api-access-jfdlz" (OuterVolumeSpecName: "kube-api-access-jfdlz") pod "3eba811e-35d4-4d7c-9c57-96ef89c26c26" (UID: "3eba811e-35d4-4d7c-9c57-96ef89c26c26"). InnerVolumeSpecName "kube-api-access-jfdlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:57.252328 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.252291    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eba811e-35d4-4d7c-9c57-96ef89c26c26-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:57.252328 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.252325    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfdlz\" (UniqueName: \"kubernetes.io/projected/3eba811e-35d4-4d7c-9c57-96ef89c26c26-kube-api-access-jfdlz\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:57.252498 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.252340    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eba811e-35d4-4d7c-9c57-96ef89c26c26-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:40:57.807324 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.807289    2578 generic.go:356] "Generic (PLEG): container finished" podID="3eba811e-35d4-4d7c-9c57-96ef89c26c26" containerID="921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75" exitCode=0
Sep 07 11:40:57.807500 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.807330    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"
Sep 07 11:40:57.807500 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.807360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6" event={"ID":"3eba811e-35d4-4d7c-9c57-96ef89c26c26","Type":"ContainerDied","Data":"921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75"}
Sep 07 11:40:57.807500 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.807396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6" event={"ID":"3eba811e-35d4-4d7c-9c57-96ef89c26c26","Type":"ContainerDied","Data":"7d66bad21327671c9551a68241c4a62a255e228b97da30590b16816d0b6441f5"}
Sep 07 11:40:57.807500 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.807417    2578 scope.go:117] "RemoveContainer" containerID="921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75"
Sep 07 11:40:57.817368 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.817349    2578 scope.go:117] "RemoveContainer" containerID="921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75"
Sep 07 11:40:57.817701 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:40:57.817679    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75\": container with ID starting with 921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75 not found: ID does not exist" containerID="921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75"
Sep 07 11:40:57.817774 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.817708    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75"} err="failed to get container status \"921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75\": rpc error: code = NotFound desc = could not find container \"921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75\": container with ID starting with 921a90dca9ac1c9c56440c778004749909030dc5ab7cefbfbe571fbbc4b0ef75 not found: ID does not exist"
Sep 07 11:40:57.831009 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.830985    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"]
Sep 07 11:40:57.834550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:57.834521    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ef61c5/test-model-ef61c5-kserve-6bd9b7fbc7-zkkv6"]
Sep 07 11:40:58.961644 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:58.961611    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eba811e-35d4-4d7c-9c57-96ef89c26c26" path="/var/lib/kubelet/pods/3eba811e-35d4-4d7c-9c57-96ef89c26c26/volumes"
Sep 07 11:40:59.255209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.255132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:59.314210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.314178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:59.586009 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.585912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:59.731981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.731952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:40:59.956278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956244    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"]
Sep 07 11:40:59.956652 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956640    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eba811e-35d4-4d7c-9c57-96ef89c26c26" containerName="main"
Sep 07 11:40:59.956711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956654    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eba811e-35d4-4d7c-9c57-96ef89c26c26" containerName="main"
Sep 07 11:40:59.956711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d69976-10f6-47db-9cae-10327562842f" containerName="main"
Sep 07 11:40:59.956711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956673    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d69976-10f6-47db-9cae-10327562842f" containerName="main"
Sep 07 11:40:59.956711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956681    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564cf7c7-3582-413b-a204-d62bdd555b84" containerName="main"
Sep 07 11:40:59.956711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956687    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="564cf7c7-3582-413b-a204-d62bdd555b84" containerName="main"
Sep 07 11:40:59.956711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956695    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7442b622-342b-4670-af53-939774b2a0f9" containerName="revoke-keys"
Sep 07 11:40:59.956711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956700    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7442b622-342b-4670-af53-939774b2a0f9" containerName="revoke-keys"
Sep 07 11:40:59.956911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956772    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7442b622-342b-4670-af53-939774b2a0f9" containerName="revoke-keys"
Sep 07 11:40:59.956911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eba811e-35d4-4d7c-9c57-96ef89c26c26" containerName="main"
Sep 07 11:40:59.956911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956787    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d69976-10f6-47db-9cae-10327562842f" containerName="main"
Sep 07 11:40:59.956911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.956794    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="564cf7c7-3582-413b-a204-d62bdd555b84" containerName="main"
Sep 07 11:40:59.960997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.960982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"
Sep 07 11:40:59.963496 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.963475    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:40:59.963768 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.963546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:40:59.970051 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.970020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"]
Sep 07 11:40:59.998750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:40:59.982075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c33f76-755e-4b1f-9e1b-ecbb4ce0536f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p\" (UID: \"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"
Sep 07 11:41:00.087489 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.086988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c33f76-755e-4b1f-9e1b-ecbb4ce0536f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p\" (UID: \"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"
Sep 07 11:41:00.091487 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.087824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c33f76-755e-4b1f-9e1b-ecbb4ce0536f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p\" (UID: \"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"
Sep 07 11:41:00.126416 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.126385    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:00.272750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.272659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"
Sep 07 11:41:00.408965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.408931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:00.511606 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.511574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:00.558315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.558284    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"]
Sep 07 11:41:00.559671 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:41:00.559646    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c33f76_755e_4b1f_9e1b_ecbb4ce0536f.slice/crio-ca7ba18346ae65d53989ce07c87cfb0cd518cd3e49c77387439a1fff092a1b32 WatchSource:0}: Error finding container ca7ba18346ae65d53989ce07c87cfb0cd518cd3e49c77387439a1fff092a1b32: Status 404 returned error can't find the container with id ca7ba18346ae65d53989ce07c87cfb0cd518cd3e49c77387439a1fff092a1b32
Sep 07 11:41:00.829084 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.829002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p" event={"ID":"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f","Type":"ContainerStarted","Data":"ca7ba18346ae65d53989ce07c87cfb0cd518cd3e49c77387439a1fff092a1b32"}
Sep 07 11:41:00.873200 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:00.873160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:01.836517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:01.836409    2578 generic.go:356] "Generic (PLEG): container finished" podID="78c33f76-755e-4b1f-9e1b-ecbb4ce0536f" containerID="bf4c89c28e2d7a7feca2cd4b9f99be6e37a9af02c0501b556bee6f2aed8929a7" exitCode=0
Sep 07 11:41:01.836517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:01.836487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p" event={"ID":"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f","Type":"ContainerDied","Data":"bf4c89c28e2d7a7feca2cd4b9f99be6e37a9af02c0501b556bee6f2aed8929a7"}
Sep 07 11:41:02.002230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:02.002194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:41:02.988254 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:02.988230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"
Sep 07 11:41:03.019307 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:03.019277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c33f76-755e-4b1f-9e1b-ecbb4ce0536f-maas-api-service-ca\") pod \"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f\" (UID: \"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f\") "
Sep 07 11:41:03.019869 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:03.019645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c33f76-755e-4b1f-9e1b-ecbb4ce0536f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78c33f76-755e-4b1f-9e1b-ecbb4ce0536f" (UID: "78c33f76-755e-4b1f-9e1b-ecbb4ce0536f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:41:03.019869 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:03.019786    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c33f76-755e-4b1f-9e1b-ecbb4ce0536f-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:03.850216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:03.850180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p" event={"ID":"78c33f76-755e-4b1f-9e1b-ecbb4ce0536f","Type":"ContainerDied","Data":"ca7ba18346ae65d53989ce07c87cfb0cd518cd3e49c77387439a1fff092a1b32"}
Sep 07 11:41:03.850216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:03.850206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"
Sep 07 11:41:03.850427 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:03.850233    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca7ba18346ae65d53989ce07c87cfb0cd518cd3e49c77387439a1fff092a1b32"
Sep 07 11:41:04.697650 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:04.697613    2578 patch_prober.go:28] interesting pod/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 11:41:04.698027 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:04.697674    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 11:41:06.665897 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.665856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:06.723099 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.723071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:06.870103 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.870071    2578 generic.go:356] "Generic (PLEG): container finished" podID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerID="29de278882a25632bd1a38038fbfd2df6be13ce797ddead49abf222ea22be7bd" exitCode=0
Sep 07 11:41:06.870251 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.870141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" event={"ID":"d98d72b2-96ea-4d62-a288-b18c8d7db8bd","Type":"ContainerDied","Data":"29de278882a25632bd1a38038fbfd2df6be13ce797ddead49abf222ea22be7bd"}
Sep 07 11:41:06.870575 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.870552    2578 scope.go:117] "RemoveContainer" containerID="29de278882a25632bd1a38038fbfd2df6be13ce797ddead49abf222ea22be7bd"
Sep 07 11:41:06.871648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.871620    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:06.871745 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.871699    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:06.871745 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:06.871734    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:07.322409 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:07.322334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:07.882797 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:07.882751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" event={"ID":"d98d72b2-96ea-4d62-a288-b18c8d7db8bd","Type":"ContainerStarted","Data":"ca8f91e408d6099cde8e7b0f717971ff292f1a8593aba995de36be558980f3c0"}
Sep 07 11:41:08.297024 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:08.296911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:41:08.297235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:08.297195    2578 patch_prober.go:28] interesting pod/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 07 11:41:08.297372 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:08.297256    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 07 11:41:09.296650 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:09.296610    2578 patch_prober.go:28] interesting pod/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 07 11:41:09.297068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:09.296680    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 07 11:41:10.120157 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.120120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:10.296904 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.296864    2578 patch_prober.go:28] interesting pod/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 07 11:41:10.297263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.296920    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 07 11:41:10.697363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.697327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:10.722639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.722578    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"]
Sep 07 11:41:10.723091 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.723052    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc" podUID="6d350635-d3ae-4820-86d2-476e05b079a7" containerName="payload-pre-processing" containerID="cri-o://1e4823a73b501bc3c09cb1ba08a069c3cf22ad344b3a463e72676945f187c397" gracePeriod=30
Sep 07 11:41:10.795892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.795852    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:10.881661 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.881617    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"]
Sep 07 11:41:10.886486 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.885239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:10.895134 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.895109    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef61c5-95c2p"]
Sep 07 11:41:10.907266 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.907243    2578 generic.go:356] "Generic (PLEG): container finished" podID="6d350635-d3ae-4820-86d2-476e05b079a7" containerID="1e4823a73b501bc3c09cb1ba08a069c3cf22ad344b3a463e72676945f187c397" exitCode=0
Sep 07 11:41:10.907373 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.907321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc" event={"ID":"6d350635-d3ae-4820-86d2-476e05b079a7","Type":"ContainerDied","Data":"1e4823a73b501bc3c09cb1ba08a069c3cf22ad344b3a463e72676945f187c397"}
Sep 07 11:41:10.945995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.945956    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:10.960273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:10.960212    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c33f76-755e-4b1f-9e1b-ecbb4ce0536f" path="/var/lib/kubelet/pods/78c33f76-755e-4b1f-9e1b-ecbb4ce0536f/volumes"
Sep 07 11:41:11.019147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.019126    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:41:11.105852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.105816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m9m9\" (UniqueName: \"kubernetes.io/projected/6d350635-d3ae-4820-86d2-476e05b079a7-kube-api-access-4m9m9\") pod \"6d350635-d3ae-4820-86d2-476e05b079a7\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") "
Sep 07 11:41:11.106061 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.105875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d350635-d3ae-4820-86d2-476e05b079a7-plugins-config-volume\") pod \"6d350635-d3ae-4820-86d2-476e05b079a7\" (UID: \"6d350635-d3ae-4820-86d2-476e05b079a7\") "
Sep 07 11:41:11.106271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.106240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d350635-d3ae-4820-86d2-476e05b079a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d350635-d3ae-4820-86d2-476e05b079a7" (UID: "6d350635-d3ae-4820-86d2-476e05b079a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:41:11.107965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.107941    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d350635-d3ae-4820-86d2-476e05b079a7-kube-api-access-4m9m9" (OuterVolumeSpecName: "kube-api-access-4m9m9") pod "6d350635-d3ae-4820-86d2-476e05b079a7" (UID: "6d350635-d3ae-4820-86d2-476e05b079a7"). InnerVolumeSpecName "kube-api-access-4m9m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:41:11.207145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.207112    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d350635-d3ae-4820-86d2-476e05b079a7-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:11.207145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.207139    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m9m9\" (UniqueName: \"kubernetes.io/projected/6d350635-d3ae-4820-86d2-476e05b079a7-kube-api-access-4m9m9\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:11.292520 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.292395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:11.296323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.296295    2578 patch_prober.go:28] interesting pod/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 07 11:41:11.296447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.296355    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 07 11:41:11.914520 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.914490    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"
Sep 07 11:41:11.914934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.914489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc" event={"ID":"6d350635-d3ae-4820-86d2-476e05b079a7","Type":"ContainerDied","Data":"bb64b4220774de7a7f2ca3334349d93812ee81214101b3a76c86dd75e1191e06"}
Sep 07 11:41:11.914934 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.914618    2578 scope.go:117] "RemoveContainer" containerID="1e4823a73b501bc3c09cb1ba08a069c3cf22ad344b3a463e72676945f187c397"
Sep 07 11:41:11.935806 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.935779    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"]
Sep 07 11:41:11.940085 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:11.940064    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef61c5-66dd8dc678-cfnvc"]
Sep 07 11:41:12.296669 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:12.296582    2578 patch_prober.go:28] interesting pod/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 07 11:41:12.296669 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:12.296635    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 07 11:41:12.961696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:12.961653    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d350635-d3ae-4820-86d2-476e05b079a7" path="/var/lib/kubelet/pods/6d350635-d3ae-4820-86d2-476e05b079a7/volumes"
Sep 07 11:41:13.300343 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:13.300259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:41:13.300567 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:13.300538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:41:13.301312 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:13.301297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:41:15.194508 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:15.194475    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:41:15.741128 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:15.741097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:16.905944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:16.905907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:17.354084 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:17.353986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:17.560401 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:17.560370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:17.654200 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:17.654164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:17.802107 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:17.802068    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"]
Sep 07 11:41:17.802305 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:17.802281    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy" containerID="cri-o://02e87caadb8a413d0b19bfc5153ef4bf2636458736495a423e75360806eb8c31" gracePeriod=30
Sep 07 11:41:17.825300 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:17.825275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:17.976692 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:17.976607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:18.050256 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:18.050228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:18.117732 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:18.117689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:18.181966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:18.181918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:18.404042 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:18.403987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:18.552759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:18.552725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:18.724144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:18.724069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:19.140877 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:19.140846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:21.858983 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:21.858940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:21.950211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:21.950156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:22.989072 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:22.989038    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerID="02e87caadb8a413d0b19bfc5153ef4bf2636458736495a423e75360806eb8c31" exitCode=0
Sep 07 11:41:22.989507 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:22.989118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" event={"ID":"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7","Type":"ContainerDied","Data":"02e87caadb8a413d0b19bfc5153ef4bf2636458736495a423e75360806eb8c31"}
Sep 07 11:41:22.989507 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:22.989194    2578 scope.go:117] "RemoveContainer" containerID="e277d4419979a00cf9b7fc299210022b0887703ebbd3f560029ad27c3ae62ee4"
Sep 07 11:41:23.006611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.006543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:23.062557 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.062531    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:41:23.090203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.090133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:23.215456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-socket\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215657 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-certs\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215657 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-credential-socket\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215657 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215600    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-data\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215657 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-podinfo\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215870 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bqbq\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-kube-api-access-6bqbq\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215870 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-envoy\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215870 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215713    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istiod-ca-cert\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.215870 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:23.215870 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-token\") pod \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\" (UID: \"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7\") "
Sep 07 11:41:23.216129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.215968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:23.216129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.216010    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:23.216239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.216118    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.216239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.216150    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.216239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.216153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:23.216239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.216165    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.216239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.216201    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:41:23.217928 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.217902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:41:23.217928 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.217916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-kube-api-access-6bqbq" (OuterVolumeSpecName: "kube-api-access-6bqbq") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "kube-api-access-6bqbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:41:23.218294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.218273    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:41:23.218294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.218275    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" (UID: "1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:23.317042 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.317007    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.317042 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.317035    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.317042 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.317045    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.317263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.317053    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bqbq\" (UniqueName: \"kubernetes.io/projected/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-kube-api-access-6bqbq\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.317263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.317063    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.317263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.317070    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:23.634831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.634792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:23.727184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.727141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:23.907459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.907373    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:41:23.997446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.997417    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"
Sep 07 11:41:23.997964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.997450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6" event={"ID":"1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7","Type":"ContainerDied","Data":"eeea96128da6d54a58854737919379d7c6dd39cd2f3e8894b536d7b9fa350b0f"}
Sep 07 11:41:23.997964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:23.997518    2578 scope.go:117] "RemoveContainer" containerID="02e87caadb8a413d0b19bfc5153ef4bf2636458736495a423e75360806eb8c31"
Sep 07 11:41:24.021305 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:24.021278    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"]
Sep 07 11:41:24.024603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:24.024577    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef61c5-openshift-default-6f5f4cffb4-zg6l6"]
Sep 07 11:41:24.625436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:24.625386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:24.960357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:24.960280    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" path="/var/lib/kubelet/pods/1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7/volumes"
Sep 07 11:41:25.650503 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:25.650454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:25.753571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:25.753527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:25.898837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:25.898798    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"]
Sep 07 11:41:25.899071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:25.899044    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" containerID="cri-o://ca8f91e408d6099cde8e7b0f717971ff292f1a8593aba995de36be558980f3c0" gracePeriod=30
Sep 07 11:41:25.922097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:25.922008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:26.310968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:26.310864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:26.412784 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:26.412723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:26.523034 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:26.523001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:28.303285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:28.303244    2578 patch_prober.go:28] interesting pod/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 11:41:28.303675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:28.303311    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 11:41:29.890799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:29.890751    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:41:31.045118 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.045077    2578 generic.go:356] "Generic (PLEG): container finished" podID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerID="ca8f91e408d6099cde8e7b0f717971ff292f1a8593aba995de36be558980f3c0" exitCode=0
Sep 07 11:41:31.045523 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.045144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" event={"ID":"d98d72b2-96ea-4d62-a288-b18c8d7db8bd","Type":"ContainerDied","Data":"ca8f91e408d6099cde8e7b0f717971ff292f1a8593aba995de36be558980f3c0"}
Sep 07 11:41:31.045523 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.045206    2578 scope.go:117] "RemoveContainer" containerID="29de278882a25632bd1a38038fbfd2df6be13ce797ddead49abf222ea22be7bd"
Sep 07 11:41:31.145663 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.145638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:41:31.295245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-credential-socket\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-envoy\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295218    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-socket\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295237    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwk4l\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-kube-api-access-dwk4l\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295264    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-podinfo\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-data\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-token\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295378    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istiod-ca-cert\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295426    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-certs\") pod \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\" (UID: \"d98d72b2-96ea-4d62-a288-b18c8d7db8bd\") "
Sep 07 11:41:31.295637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:31.295637 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:31.295987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:31.295987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295842    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.295987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295860    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.295987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295875    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.295987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:41:31.295987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.295900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:31.297549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.297514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:41:31.297675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.297605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:41:31.297722 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.297679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-kube-api-access-dwk4l" (OuterVolumeSpecName: "kube-api-access-dwk4l") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "kube-api-access-dwk4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:41:31.297722 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.297680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "d98d72b2-96ea-4d62-a288-b18c8d7db8bd" (UID: "d98d72b2-96ea-4d62-a288-b18c8d7db8bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:41:31.397306 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.397271    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.397306 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.397302    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwk4l\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-kube-api-access-dwk4l\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.397306 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.397313    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.397570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.397322    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.397570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.397330    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:31.397570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:31.397337    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98d72b2-96ea-4d62-a288-b18c8d7db8bd-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:41:32.053504 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:32.053448    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"
Sep 07 11:41:32.053914 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:32.053499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25" event={"ID":"d98d72b2-96ea-4d62-a288-b18c8d7db8bd","Type":"ContainerDied","Data":"07ba1b4341fbcb378623fe1a5fa021febbe7389e3d18be882fa5c3b14dbe7c79"}
Sep 07 11:41:32.053914 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:32.053564    2578 scope.go:117] "RemoveContainer" containerID="ca8f91e408d6099cde8e7b0f717971ff292f1a8593aba995de36be558980f3c0"
Sep 07 11:41:32.088109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:32.088076    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"]
Sep 07 11:41:32.095237 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:32.095199    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d6b326a7-openshift-default-6bdb644c4d-55j25"]
Sep 07 11:41:32.939179 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:32.939147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:32.960744 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:32.960700    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" path="/var/lib/kubelet/pods/d98d72b2-96ea-4d62-a288-b18c8d7db8bd/volumes"
Sep 07 11:41:33.020407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:33.020372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:33.082201 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:33.082164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:33.181436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:33.181399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:33.294919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:33.294837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:39.467891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:39.467855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:39.522535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:39.522509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:40.402363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.402309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:40.701419 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701323    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"]
Sep 07 11:41:40.701879 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701864    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy"
Sep 07 11:41:40.701945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701882    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy"
Sep 07 11:41:40.701945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c33f76-755e-4b1f-9e1b-ecbb4ce0536f" containerName="revoke-keys"
Sep 07 11:41:40.701945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701918    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c33f76-755e-4b1f-9e1b-ecbb4ce0536f" containerName="revoke-keys"
Sep 07 11:41:40.701945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701933    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d350635-d3ae-4820-86d2-476e05b079a7" containerName="payload-pre-processing"
Sep 07 11:41:40.701945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701942    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d350635-d3ae-4820-86d2-476e05b079a7" containerName="payload-pre-processing"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701951    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701959    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701970    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.701978    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.702070    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.702080    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c33f76-755e-4b1f-9e1b-ecbb4ce0536f" containerName="revoke-keys"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.702088    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.702098    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d350635-d3ae-4820-86d2-476e05b079a7" containerName="payload-pre-processing"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.702109    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy"
Sep 07 11:41:40.702209 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.702119    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98d72b2-96ea-4d62-a288-b18c8d7db8bd" containerName="istio-proxy"
Sep 07 11:41:40.707058 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.707034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.709608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.709583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-0c816d-openshift-default-dockercfg-hmmp8\""
Sep 07 11:41:40.709726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.709607    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 11:41:40.717057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.717033    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"]
Sep 07 11:41:40.730111 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.730077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:40.770989 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.770954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-certs\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.770996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-envoy\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.771019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8ql\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-kube-api-access-jp8ql\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.771049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-podinfo\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.771091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-data\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.771129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-token\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.771156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-socket\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.771225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461664cb-93d3-4e8f-a937-642f085d3f4b-istiod-ca-cert\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.771330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.771251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-credential-socket\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872142 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-envoy\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872142 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8ql\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-kube-api-access-jp8ql\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-podinfo\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-data\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-token\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-socket\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461664cb-93d3-4e8f-a937-642f085d3f4b-istiod-ca-cert\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-credential-socket\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872691 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-certs\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-data\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-certs\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872859 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-credential-socket\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.872947 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.872923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-socket\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.873152 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.873132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461664cb-93d3-4e8f-a937-642f085d3f4b-istiod-ca-cert\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.874805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.874778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-podinfo\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.874995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.874979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-envoy\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.879938 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.879913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-token\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:40.880074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:40.880052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8ql\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-kube-api-access-jp8ql\") pod \"e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") " pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:41.019334 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:41.019259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:41.193272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:41.193237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:41.295753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:41.295721    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"]
Sep 07 11:41:41.296972 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:41:41.296948    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461664cb_93d3_4e8f_a937_642f085d3f4b.slice/crio-741f99bb58011d8cb751f100fbb825572e08e46d40fbb4d353d4e1278fdfb742 WatchSource:0}: Error finding container 741f99bb58011d8cb751f100fbb825572e08e46d40fbb4d353d4e1278fdfb742: Status 404 returned error can't find the container with id 741f99bb58011d8cb751f100fbb825572e08e46d40fbb4d353d4e1278fdfb742
Sep 07 11:41:41.299143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:41.299110    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:41.299235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:41.299186    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:41.299298 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:41.299230    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:42.114949 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:42.114917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" event={"ID":"461664cb-93d3-4e8f-a937-642f085d3f4b","Type":"ContainerStarted","Data":"235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb"}
Sep 07 11:41:42.115311 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:42.114969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" event={"ID":"461664cb-93d3-4e8f-a937-642f085d3f4b","Type":"ContainerStarted","Data":"741f99bb58011d8cb751f100fbb825572e08e46d40fbb4d353d4e1278fdfb742"}
Sep 07 11:41:42.139232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:42.139185    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podStartSLOduration=2.139170238 podStartE2EDuration="2.139170238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:42.137103471 +0000 UTC m=+1133.611951646" watchObservedRunningTime="2026-09-07 11:41:42.139170238 +0000 UTC m=+1133.614018387"
Sep 07 11:41:43.020199 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:43.020169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:43.025072 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:43.025038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:43.120072 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:43.120044    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:43.120906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:43.120883    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:41:44.048553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:44.048522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:44.143861 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:44.143828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:45.627160 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:45.627118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:45.686757 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:45.686728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:46.044810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.044722    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:46.153050 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.153015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:46.248980 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.248946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:46.250687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.250663    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"]
Sep 07 11:41:46.251054 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.251039    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy"
Sep 07 11:41:46.251116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.251057    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7a5ef5-b785-4bf7-99a4-3559e7a6d7d7" containerName="istio-proxy"
Sep 07 11:41:46.254268 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.254249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.256829 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.256810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0c816d\""
Sep 07 11:41:46.256972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.256954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-0c816d-dockercfg-ncmpw\""
Sep 07 11:41:46.265718 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.265685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"]
Sep 07 11:41:46.316064 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.315999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6a9559-7549-42a1-99ed-4e087b97f16f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.316164 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.316108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknrv\" (UniqueName: \"kubernetes.io/projected/9f6a9559-7549-42a1-99ed-4e087b97f16f-kube-api-access-wknrv\") pod \"payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.417566 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.417538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6a9559-7549-42a1-99ed-4e087b97f16f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.417721 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.417676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknrv\" (UniqueName: \"kubernetes.io/projected/9f6a9559-7549-42a1-99ed-4e087b97f16f-kube-api-access-wknrv\") pod \"payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.418206 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.418187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6a9559-7549-42a1-99ed-4e087b97f16f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.426582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.426556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknrv\" (UniqueName: \"kubernetes.io/projected/9f6a9559-7549-42a1-99ed-4e087b97f16f-kube-api-access-wknrv\") pod \"payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.565952 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.565923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:46.836969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.836944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"]
Sep 07 11:41:46.839036 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:41:46.839007    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6a9559_7549_42a1_99ed_4e087b97f16f.slice/crio-1b262b326c9a4c4b48aad73dcbe48a6fdf2c9e7948c00f71a45acfb4f9cd1263 WatchSource:0}: Error finding container 1b262b326c9a4c4b48aad73dcbe48a6fdf2c9e7948c00f71a45acfb4f9cd1263: Status 404 returned error can't find the container with id 1b262b326c9a4c4b48aad73dcbe48a6fdf2c9e7948c00f71a45acfb4f9cd1263
Sep 07 11:41:46.847270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:46.847243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:47.143284 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.143245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p" event={"ID":"9f6a9559-7549-42a1-99ed-4e087b97f16f","Type":"ContainerStarted","Data":"6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a"}
Sep 07 11:41:47.143484 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.143302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p" event={"ID":"9f6a9559-7549-42a1-99ed-4e087b97f16f","Type":"ContainerStarted","Data":"1b262b326c9a4c4b48aad73dcbe48a6fdf2c9e7948c00f71a45acfb4f9cd1263"}
Sep 07 11:41:47.143484 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.143324    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:47.167326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.167280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p" podStartSLOduration=1.167264876 podStartE2EDuration="1.167264876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:47.166048825 +0000 UTC m=+1138.640896994" watchObservedRunningTime="2026-09-07 11:41:47.167264876 +0000 UTC m=+1138.642113024"
Sep 07 11:41:47.170789 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.170768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:47.453101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.453015    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"]
Sep 07 11:41:47.456802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.456773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.459681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.459661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-0f0a2f-openshift-default-dockercfg-826kf\""
Sep 07 11:41:47.471446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.471422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"]
Sep 07 11:41:47.492490 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.492367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:47.526962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.526880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17562604-4439-4954-acba-2f59a9d2bd47-istiod-ca-cert\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.526967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kl8g\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-kube-api-access-6kl8g\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.527035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-certs\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.527069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-credential-socket\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527124 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.527109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-istio-token\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.527153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-envoy\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.527185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-socket\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.527247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17562604-4439-4954-acba-2f59a9d2bd47-istio-podinfo\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.527397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.527276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-data\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.581584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.581550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:47.628609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kl8g\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-kube-api-access-6kl8g\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.628609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-certs\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.628843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-credential-socket\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.628843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-istio-token\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.628843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-envoy\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.628843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-socket\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.629055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17562604-4439-4954-acba-2f59a9d2bd47-istio-podinfo\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.629055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-data\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.629055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.628925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17562604-4439-4954-acba-2f59a9d2bd47-istiod-ca-cert\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.629774 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.629741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17562604-4439-4954-acba-2f59a9d2bd47-istiod-ca-cert\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.630004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.629978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-socket\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.630119 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.630093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-certs\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.630119 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.630105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-credential-socket\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.630394 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.630368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-data\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.632670 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.632643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-envoy\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.632775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.632757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17562604-4439-4954-acba-2f59a9d2bd47-istio-podinfo\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.637945 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.637910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kl8g\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-kube-api-access-6kl8g\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.638036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.637982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-istio-token\") pod \"e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") " pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:47.767223 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:47.767142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:48.045982 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.045952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"]
Sep 07 11:41:48.048716 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:41:48.048689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17562604_4439_4954_acba_2f59a9d2bd47.slice/crio-41723ad8f90cf523c8231bb5b756821459f8a0c7b4737e71d2df1d701af40fb3 WatchSource:0}: Error finding container 41723ad8f90cf523c8231bb5b756821459f8a0c7b4737e71d2df1d701af40fb3: Status 404 returned error can't find the container with id 41723ad8f90cf523c8231bb5b756821459f8a0c7b4737e71d2df1d701af40fb3
Sep 07 11:41:48.050842 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.050809    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:48.050935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.050877    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:48.050935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.050901    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:41:48.151326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.151290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" event={"ID":"17562604-4439-4954-acba-2f59a9d2bd47","Type":"ContainerStarted","Data":"cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9"}
Sep 07 11:41:48.151530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.151403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" event={"ID":"17562604-4439-4954-acba-2f59a9d2bd47","Type":"ContainerStarted","Data":"41723ad8f90cf523c8231bb5b756821459f8a0c7b4737e71d2df1d701af40fb3"}
Sep 07 11:41:48.176376 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.176319    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podStartSLOduration=1.176305266 podStartE2EDuration="1.176305266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:48.173000312 +0000 UTC m=+1139.647848487" watchObservedRunningTime="2026-09-07 11:41:48.176305266 +0000 UTC m=+1139.651153415"
Sep 07 11:41:48.767924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:48.767889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:49.773548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:49.773515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:50.159873 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:50.159832    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:50.162368 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:50.162342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:50.163323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:50.163301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:41:50.253598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:50.253521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:50.545081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:50.544995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:41:51.331296 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:51.331240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:51.431403 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:51.431369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:51.532819 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:51.532714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:51.909432 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:51.909399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:52.009933 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.009892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:41:52.057419 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.057391    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"]
Sep 07 11:41:52.062534 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.062513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.064972 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.064947    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0f0a2f\""
Sep 07 11:41:52.065073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.064970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0f0a2f-dockercfg-tg9sm\""
Sep 07 11:41:52.074452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.074429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"]
Sep 07 11:41:52.168715 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.168633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnlw5\" (UniqueName: \"kubernetes.io/projected/5d8287c5-6574-4721-be96-1473fd2ef885-kube-api-access-dnlw5\") pod \"payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.168715 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.168698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8287c5-6574-4721-be96-1473fd2ef885-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.269696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.269661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnlw5\" (UniqueName: \"kubernetes.io/projected/5d8287c5-6574-4721-be96-1473fd2ef885-kube-api-access-dnlw5\") pod \"payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.269872 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.269824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8287c5-6574-4721-be96-1473fd2ef885-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.270500 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.270449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8287c5-6574-4721-be96-1473fd2ef885-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.278108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.278090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnlw5\" (UniqueName: \"kubernetes.io/projected/5d8287c5-6574-4721-be96-1473fd2ef885-kube-api-access-dnlw5\") pod \"payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.373311 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.373282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:52.653891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:52.653863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"]
Sep 07 11:41:52.655854 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:41:52.655829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8287c5_6574_4721_be96_1473fd2ef885.slice/crio-730013aa72a7d12bc8219ee6b6fac17967db94dbeb8bbfc6611e8b0f3d0e5db9 WatchSource:0}: Error finding container 730013aa72a7d12bc8219ee6b6fac17967db94dbeb8bbfc6611e8b0f3d0e5db9: Status 404 returned error can't find the container with id 730013aa72a7d12bc8219ee6b6fac17967db94dbeb8bbfc6611e8b0f3d0e5db9
Sep 07 11:41:53.187777 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:53.187726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb" event={"ID":"5d8287c5-6574-4721-be96-1473fd2ef885","Type":"ContainerStarted","Data":"32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed"}
Sep 07 11:41:53.187998 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:53.187804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb" event={"ID":"5d8287c5-6574-4721-be96-1473fd2ef885","Type":"ContainerStarted","Data":"730013aa72a7d12bc8219ee6b6fac17967db94dbeb8bbfc6611e8b0f3d0e5db9"}
Sep 07 11:41:53.187998 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:53.187986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:41:53.208918 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:53.208795    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb" podStartSLOduration=1.208774607 podStartE2EDuration="1.208774607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:53.208125958 +0000 UTC m=+1144.682974144" watchObservedRunningTime="2026-09-07 11:41:53.208774607 +0000 UTC m=+1144.683622761"
Sep 07 11:41:58.153161 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:58.153129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:41:59.161669 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:59.161622    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 11:41:59.162230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:41:59.161714    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 11:42:01.393180 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:01.393133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:42:02.256349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:02.256257    2578 generic.go:356] "Generic (PLEG): container finished" podID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerID="235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb" exitCode=0
Sep 07 11:42:02.256349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:02.256327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" event={"ID":"461664cb-93d3-4e8f-a937-642f085d3f4b","Type":"ContainerDied","Data":"235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb"}
Sep 07 11:42:02.256759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:02.256746    2578 scope.go:117] "RemoveContainer" containerID="235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb"
Sep 07 11:42:02.257924 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:02.257900    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:42:02.257988 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:02.257962    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:42:02.258029 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:02.257992    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:42:03.263752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:03.263717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" event={"ID":"461664cb-93d3-4e8f-a937-642f085d3f4b","Type":"ContainerStarted","Data":"1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53"}
Sep 07 11:42:04.019784 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:04.019739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:42:04.020007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:04.019983    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 11:42:04.020060 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:04.020035    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 11:42:04.200133 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:04.200098    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:42:05.020045 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:05.019995    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 11:42:05.020563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:05.020078    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 11:42:06.020247 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:06.020206    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 11:42:06.020749 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:06.020274    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 11:42:06.204704 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:06.204659    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 11:42:06.204883 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:06.204722    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 11:42:06.544332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:06.544284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:07.020238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.020192    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 11:42:07.020403 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.020260    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 11:42:07.300210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.300133    2578 generic.go:356] "Generic (PLEG): container finished" podID="17562604-4439-4954-acba-2f59a9d2bd47" containerID="cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9" exitCode=0
Sep 07 11:42:07.300406 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.300216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" event={"ID":"17562604-4439-4954-acba-2f59a9d2bd47","Type":"ContainerDied","Data":"cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9"}
Sep 07 11:42:07.300723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.300705    2578 scope.go:117] "RemoveContainer" containerID="cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9"
Sep 07 11:42:07.301897 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.301869    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:42:07.301963 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.301945    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:42:07.302006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:07.301978    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:42:08.020669 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:08.020631    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 11:42:08.021074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:08.020687    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 11:42:08.317017 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:08.316921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" event={"ID":"17562604-4439-4954-acba-2f59a9d2bd47","Type":"ContainerStarted","Data":"f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088"}
Sep 07 11:42:08.767429 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:08.767395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:42:08.767632 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:08.767612    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 07 11:42:08.767680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:08.767659    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 07 11:42:09.023434 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.023360    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:42:09.023802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.023677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:42:09.024271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.024251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:42:09.038974 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.038951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:09.393893 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.393863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:09.714839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.714744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:09.768375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.768340    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 07 11:42:09.768560 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:09.768403    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 07 11:42:10.174695 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:10.174652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:10.768074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:10.768028    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 07 11:42:10.768405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:10.768115    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 07 11:42:11.767912 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:11.767834    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 07 11:42:11.767912 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:11.767888    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 07 11:42:12.384750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:12.384698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:12.768427 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:12.768337    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 07 11:42:12.768427 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:12.768393    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 07 11:42:13.225955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:13.225917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:13.315413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:13.315376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:13.771235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:13.771206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:42:13.771634 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:13.771615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:42:13.772177 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:13.772155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:42:14.544978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:14.544939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:14.644927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:14.644889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:14.704439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:14.704407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:15.028009 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.027974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:15.101949 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.101920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:15.125729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.125699    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"]
Sep 07 11:42:15.128240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.128224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.130794 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.130762    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2195e0\""
Sep 07 11:42:15.130909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.130862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2195e0-dockercfg-txr9n\""
Sep 07 11:42:15.141230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.141208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"]
Sep 07 11:42:15.182920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.182893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:15.296617 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.296537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.296760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.296674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hktp\" (UniqueName: \"kubernetes.io/projected/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-kube-api-access-8hktp\") pod \"payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.397345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.397301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hktp\" (UniqueName: \"kubernetes.io/projected/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-kube-api-access-8hktp\") pod \"payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.397549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.397365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.398068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.398041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.405438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.405412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hktp\" (UniqueName: \"kubernetes.io/projected/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-kube-api-access-8hktp\") pod \"payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.438787 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.438763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:15.711260 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:15.711231    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"]
Sep 07 11:42:15.713544 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:15.713519    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0413d4_b232_4a97_b0e1_01c86aa2adb8.slice/crio-863087188a52945f04077a39abc3ea9fc5f0517873fa0e90429b60723efc8391 WatchSource:0}: Error finding container 863087188a52945f04077a39abc3ea9fc5f0517873fa0e90429b60723efc8391: Status 404 returned error can't find the container with id 863087188a52945f04077a39abc3ea9fc5f0517873fa0e90429b60723efc8391
Sep 07 11:42:16.368654 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:16.368565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6" event={"ID":"5d0413d4-b232-4a97-b0e1-01c86aa2adb8","Type":"ContainerStarted","Data":"ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da"}
Sep 07 11:42:16.368654 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:16.368626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6" event={"ID":"5d0413d4-b232-4a97-b0e1-01c86aa2adb8","Type":"ContainerStarted","Data":"863087188a52945f04077a39abc3ea9fc5f0517873fa0e90429b60723efc8391"}
Sep 07 11:42:16.369276 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:16.368795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:16.386750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:16.386703    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6" podStartSLOduration=1.3866885469999999 podStartE2EDuration="1.386688547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:42:16.385893015 +0000 UTC m=+1167.860741186" watchObservedRunningTime="2026-09-07 11:42:16.386688547 +0000 UTC m=+1167.861536696"
Sep 07 11:42:20.806022 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:20.805989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:42:27.377411 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:27.377376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:42:32.410595 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.410561    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"]
Sep 07 11:42:32.423710 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.423680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.424502 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.424452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"]
Sep 07 11:42:32.427412 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.427388    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"auth-test-model-6b944b-kserve-self-signed-certs\""
Sep 07 11:42:32.427541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.427449    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"openshift-service-ca.crt\""
Sep 07 11:42:32.427541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.427499    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"kube-root-ca.crt\""
Sep 07 11:42:32.429066 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.429042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"default-dockercfg-mbjkb\""
Sep 07 11:42:32.436255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.436233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:32.443960 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.443933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr7pk\" (UniqueName: \"kubernetes.io/projected/6b3e2cdd-1090-43c3-bef4-d74b409e3279-kube-api-access-sr7pk\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.444053 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.443998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tls-certs\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.444099 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.444054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-home\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.444132 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.444117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tmp-dir\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.444218 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.444196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-model-cache\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.444272 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.444256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-dshm\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.545563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.545526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-home\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.545737 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.545571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tmp-dir\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.545737 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.545600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-model-cache\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.545737 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.545624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-dshm\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.545737 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.545659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr7pk\" (UniqueName: \"kubernetes.io/projected/6b3e2cdd-1090-43c3-bef4-d74b409e3279-kube-api-access-sr7pk\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.545737 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.545711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tls-certs\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.546020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.545939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-home\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.546020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.546006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tmp-dir\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.546125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.546024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-model-cache\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.547984 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.547953    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-dshm\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.548207 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.548190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tls-certs\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.554607 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.554581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr7pk\" (UniqueName: \"kubernetes.io/projected/6b3e2cdd-1090-43c3-bef4-d74b409e3279-kube-api-access-sr7pk\") pod \"auth-test-model-6b944b-kserve-666cb6f647-h7kkn\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:32.735174 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:32.735091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:33.010425 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:33.010399    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"]
Sep 07 11:42:33.012077 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:33.012043    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b3e2cdd_1090_43c3_bef4_d74b409e3279.slice/crio-1fa2fead157d8c9e4a2d2896b19fc9458d992240f1072398d3af5537a09f1547 WatchSource:0}: Error finding container 1fa2fead157d8c9e4a2d2896b19fc9458d992240f1072398d3af5537a09f1547: Status 404 returned error can't find the container with id 1fa2fead157d8c9e4a2d2896b19fc9458d992240f1072398d3af5537a09f1547
Sep 07 11:42:33.467388 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:33.467354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn" event={"ID":"6b3e2cdd-1090-43c3-bef4-d74b409e3279","Type":"ContainerStarted","Data":"13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6"}
Sep 07 11:42:33.467821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:33.467395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn" event={"ID":"6b3e2cdd-1090-43c3-bef4-d74b409e3279","Type":"ContainerStarted","Data":"1fa2fead157d8c9e4a2d2896b19fc9458d992240f1072398d3af5537a09f1547"}
Sep 07 11:42:37.895824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:37.895759    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn" podStartSLOduration=5.895740499 podStartE2EDuration="5.895740499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:42:33.485037457 +0000 UTC m=+1184.959885628" watchObservedRunningTime="2026-09-07 11:42:37.895740499 +0000 UTC m=+1189.370588647"
Sep 07 11:42:37.898059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:37.898030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:42:38.186990 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:38.186908    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:38.872247 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:38.872212    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:38.875926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:38.875907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:38.879007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:38.878987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhmxq\""
Sep 07 11:42:38.889687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:38.889665    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:39.005264 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:39.005229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtlk\" (UniqueName: \"kubernetes.io/projected/f41b0c2c-1994-4a64-b17f-9b053ac0263d-kube-api-access-zvtlk\") pod \"test-curl-7221-140711668926336\" (UID: \"f41b0c2c-1994-4a64-b17f-9b053ac0263d\") " pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:39.106719 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:39.106681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtlk\" (UniqueName: \"kubernetes.io/projected/f41b0c2c-1994-4a64-b17f-9b053ac0263d-kube-api-access-zvtlk\") pod \"test-curl-7221-140711668926336\" (UID: \"f41b0c2c-1994-4a64-b17f-9b053ac0263d\") " pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:39.115170 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:39.115136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtlk\" (UniqueName: \"kubernetes.io/projected/f41b0c2c-1994-4a64-b17f-9b053ac0263d-kube-api-access-zvtlk\") pod \"test-curl-7221-140711668926336\" (UID: \"f41b0c2c-1994-4a64-b17f-9b053ac0263d\") " pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:39.186249 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:39.186175    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:39.457976 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:39.457953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:39.459710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:39.459681    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41b0c2c_1994_4a64_b17f_9b053ac0263d.slice/crio-16f4eb880ff0758fc9eaf515edec019d334f996b0f53883bd3f9866d19e4ee31 WatchSource:0}: Error finding container 16f4eb880ff0758fc9eaf515edec019d334f996b0f53883bd3f9866d19e4ee31: Status 404 returned error can't find the container with id 16f4eb880ff0758fc9eaf515edec019d334f996b0f53883bd3f9866d19e4ee31
Sep 07 11:42:39.461891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:39.461870    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:42:39.504010 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:39.503979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926336" event={"ID":"f41b0c2c-1994-4a64-b17f-9b053ac0263d","Type":"ContainerStarted","Data":"16f4eb880ff0758fc9eaf515edec019d334f996b0f53883bd3f9866d19e4ee31"}
Sep 07 11:42:40.512217 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:40.512179    2578 generic.go:356] "Generic (PLEG): container finished" podID="f41b0c2c-1994-4a64-b17f-9b053ac0263d" containerID="c17c002a306cb6d8b9dd14938965821b1d887fb86f9482fa97650de088f083c6" exitCode=0
Sep 07 11:42:40.512594 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:40.512244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926336" event={"ID":"f41b0c2c-1994-4a64-b17f-9b053ac0263d","Type":"ContainerDied","Data":"c17c002a306cb6d8b9dd14938965821b1d887fb86f9482fa97650de088f083c6"}
Sep 07 11:42:41.651837 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.651810    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:41.672389 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.672367    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7221-140711668926336_f41b0c2c-1994-4a64-b17f-9b053ac0263d/test-curl-7221-140711668926336/0.log"
Sep 07 11:42:41.696290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.696269    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:41.700097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.700073    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:41.729604 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.729581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvtlk\" (UniqueName: \"kubernetes.io/projected/f41b0c2c-1994-4a64-b17f-9b053ac0263d-kube-api-access-zvtlk\") pod \"f41b0c2c-1994-4a64-b17f-9b053ac0263d\" (UID: \"f41b0c2c-1994-4a64-b17f-9b053ac0263d\") "
Sep 07 11:42:41.731587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.731566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41b0c2c-1994-4a64-b17f-9b053ac0263d-kube-api-access-zvtlk" (OuterVolumeSpecName: "kube-api-access-zvtlk") pod "f41b0c2c-1994-4a64-b17f-9b053ac0263d" (UID: "f41b0c2c-1994-4a64-b17f-9b053ac0263d"). InnerVolumeSpecName "kube-api-access-zvtlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:42:41.825781 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.825710    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:41.826104 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.826091    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41b0c2c-1994-4a64-b17f-9b053ac0263d" containerName="test-curl-7221-140711668926336"
Sep 07 11:42:41.826155 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.826106    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41b0c2c-1994-4a64-b17f-9b053ac0263d" containerName="test-curl-7221-140711668926336"
Sep 07 11:42:41.826194 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.826166    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41b0c2c-1994-4a64-b17f-9b053ac0263d" containerName="test-curl-7221-140711668926336"
Sep 07 11:42:41.829262 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.829246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:41.830603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.830580    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvtlk\" (UniqueName: \"kubernetes.io/projected/f41b0c2c-1994-4a64-b17f-9b053ac0263d-kube-api-access-zvtlk\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:42:41.837188 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.837156    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:41.931330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:41.931302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgm2l\" (UniqueName: \"kubernetes.io/projected/dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa-kube-api-access-jgm2l\") pod \"test-curl-7221-140711668926336\" (UID: \"dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa\") " pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:42.031752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.031725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgm2l\" (UniqueName: \"kubernetes.io/projected/dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa-kube-api-access-jgm2l\") pod \"test-curl-7221-140711668926336\" (UID: \"dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa\") " pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:42.040619 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.040595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgm2l\" (UniqueName: \"kubernetes.io/projected/dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa-kube-api-access-jgm2l\") pod \"test-curl-7221-140711668926336\" (UID: \"dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa\") " pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:42.146104 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.146080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:42.416301 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.416276    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:42.416948 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:42.416923    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbb3c9b1_ca23_421d_a5a9_73cd3c0600fa.slice/crio-43f939513bf978c2f2667197c9c3aafc167655162dacf67cf67f8088ca37946c WatchSource:0}: Error finding container 43f939513bf978c2f2667197c9c3aafc167655162dacf67cf67f8088ca37946c: Status 404 returned error can't find the container with id 43f939513bf978c2f2667197c9c3aafc167655162dacf67cf67f8088ca37946c
Sep 07 11:42:42.525541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.525506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926336" event={"ID":"dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa","Type":"ContainerStarted","Data":"43f939513bf978c2f2667197c9c3aafc167655162dacf67cf67f8088ca37946c"}
Sep 07 11:42:42.526713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.526691    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:42.526817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.526712    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f4eb880ff0758fc9eaf515edec019d334f996b0f53883bd3f9866d19e4ee31"
Sep 07 11:42:42.538918 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.538893    2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7221-140711668926336" oldPodUID="f41b0c2c-1994-4a64-b17f-9b053ac0263d" podUID="dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa"
Sep 07 11:42:42.735727 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.735651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:42.735727 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.735695    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:42.747759 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.747734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:42.959955 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:42.959932    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41b0c2c-1994-4a64-b17f-9b053ac0263d" path="/var/lib/kubelet/pods/f41b0c2c-1994-4a64-b17f-9b053ac0263d/volumes"
Sep 07 11:42:43.533869 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:43.533836    2578 generic.go:356] "Generic (PLEG): container finished" podID="dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa" containerID="bc3cb54f83edccc47bacf3ba8a6de9211171fd2467c332d590986abc9ab60b26" exitCode=0
Sep 07 11:42:43.534044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:43.533922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926336" event={"ID":"dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa","Type":"ContainerDied","Data":"bc3cb54f83edccc47bacf3ba8a6de9211171fd2467c332d590986abc9ab60b26"}
Sep 07 11:42:43.545270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:43.545242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:42:44.671582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.671560    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:44.691799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.691769    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7221-140711668926336_dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa/test-curl-7221-140711668926336/0.log"
Sep 07 11:42:44.714694 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.714664    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:44.718437 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.718413    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7221-140711668926336"]
Sep 07 11:42:44.757360 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.757335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgm2l\" (UniqueName: \"kubernetes.io/projected/dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa-kube-api-access-jgm2l\") pod \"dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa\" (UID: \"dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa\") "
Sep 07 11:42:44.759431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.759409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa-kube-api-access-jgm2l" (OuterVolumeSpecName: "kube-api-access-jgm2l") pod "dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa" (UID: "dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa"). InnerVolumeSpecName "kube-api-access-jgm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:42:44.849935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.849866    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:44.850243 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.850230    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa" containerName="test-curl-7221-140711668926336"
Sep 07 11:42:44.850285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.850244    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa" containerName="test-curl-7221-140711668926336"
Sep 07 11:42:44.850331 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.850323    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa" containerName="test-curl-7221-140711668926336"
Sep 07 11:42:44.853379 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.853365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:44.858226 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.858201    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgm2l\" (UniqueName: \"kubernetes.io/projected/dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa-kube-api-access-jgm2l\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:42:44.859265 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.859242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:44.958629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.958601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgg8w\" (UniqueName: \"kubernetes.io/projected/e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a-kube-api-access-jgg8w\") pod \"test-curl-7221-140711668926912\" (UID: \"e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a\") " pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:44.963607 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:44.961317    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa" path="/var/lib/kubelet/pods/dbb3c9b1-ca23-421d-a5a9-73cd3c0600fa/volumes"
Sep 07 11:42:45.059070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:45.059039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgg8w\" (UniqueName: \"kubernetes.io/projected/e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a-kube-api-access-jgg8w\") pod \"test-curl-7221-140711668926912\" (UID: \"e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a\") " pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:45.067589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:45.067560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgg8w\" (UniqueName: \"kubernetes.io/projected/e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a-kube-api-access-jgg8w\") pod \"test-curl-7221-140711668926912\" (UID: \"e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a\") " pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:45.163997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:45.163974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:45.434994 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:45.434961    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:45.437710 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:45.437681    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5d5924c_aff9_4c87_bcc1_5a2d1aa7368a.slice/crio-0ccbd07ef1a1a5ac96a743fd5dded13b87c08c880fd6aa58d167055b11c3020f WatchSource:0}: Error finding container 0ccbd07ef1a1a5ac96a743fd5dded13b87c08c880fd6aa58d167055b11c3020f: Status 404 returned error can't find the container with id 0ccbd07ef1a1a5ac96a743fd5dded13b87c08c880fd6aa58d167055b11c3020f
Sep 07 11:42:45.548270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:45.548230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926912" event={"ID":"e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a","Type":"ContainerStarted","Data":"0ccbd07ef1a1a5ac96a743fd5dded13b87c08c880fd6aa58d167055b11c3020f"}
Sep 07 11:42:45.549451 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:45.549432    2578 scope.go:117] "RemoveContainer" containerID="bc3cb54f83edccc47bacf3ba8a6de9211171fd2467c332d590986abc9ab60b26"
Sep 07 11:42:45.549569 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:45.549488    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926336"
Sep 07 11:42:46.557828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:46.557793    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a" containerID="57850b7715650bb57d3f19df3d36d746a810922785c3279eb18cfacf551a4de5" exitCode=0
Sep 07 11:42:46.558240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:46.557872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926912" event={"ID":"e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a","Type":"ContainerDied","Data":"57850b7715650bb57d3f19df3d36d746a810922785c3279eb18cfacf551a4de5"}
Sep 07 11:42:47.709036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.709014    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:47.732995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.732962    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7221-140711668926912_e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a/test-curl-7221-140711668926912/0.log"
Sep 07 11:42:47.758144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.758117    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:47.760354 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.760334    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:47.782685 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.782662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgg8w\" (UniqueName: \"kubernetes.io/projected/e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a-kube-api-access-jgg8w\") pod \"e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a\" (UID: \"e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a\") "
Sep 07 11:42:47.784722 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.784692    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a-kube-api-access-jgg8w" (OuterVolumeSpecName: "kube-api-access-jgg8w") pod "e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a" (UID: "e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a"). InnerVolumeSpecName "kube-api-access-jgg8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:42:47.883676 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.883653    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgg8w\" (UniqueName: \"kubernetes.io/projected/e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a-kube-api-access-jgg8w\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:42:47.897627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.897591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:47.897995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.897981    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a" containerName="test-curl-7221-140711668926912"
Sep 07 11:42:47.898050 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.897996    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a" containerName="test-curl-7221-140711668926912"
Sep 07 11:42:47.898089 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.898078    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a" containerName="test-curl-7221-140711668926912"
Sep 07 11:42:47.901010 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.900996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:47.910817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.910794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:47.984533 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:47.984509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd47r\" (UniqueName: \"kubernetes.io/projected/f6c70181-c1ba-4088-9b5f-41769425183a-kube-api-access-sd47r\") pod \"test-curl-7221-140711668926912\" (UID: \"f6c70181-c1ba-4088-9b5f-41769425183a\") " pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:48.085919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.085894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd47r\" (UniqueName: \"kubernetes.io/projected/f6c70181-c1ba-4088-9b5f-41769425183a-kube-api-access-sd47r\") pod \"test-curl-7221-140711668926912\" (UID: \"f6c70181-c1ba-4088-9b5f-41769425183a\") " pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:48.095054 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.095028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd47r\" (UniqueName: \"kubernetes.io/projected/f6c70181-c1ba-4088-9b5f-41769425183a-kube-api-access-sd47r\") pod \"test-curl-7221-140711668926912\" (UID: \"f6c70181-c1ba-4088-9b5f-41769425183a\") " pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:48.210893 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.210819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:48.483430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.483404    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:48.485185 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:48.485153    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6c70181_c1ba_4088_9b5f_41769425183a.slice/crio-2710e95eac618589ea4157903e312e855a604f43f3aa30ac22020d27d1598a54 WatchSource:0}: Error finding container 2710e95eac618589ea4157903e312e855a604f43f3aa30ac22020d27d1598a54: Status 404 returned error can't find the container with id 2710e95eac618589ea4157903e312e855a604f43f3aa30ac22020d27d1598a54
Sep 07 11:42:48.572693 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.572668    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:48.572817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.572714    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ccbd07ef1a1a5ac96a743fd5dded13b87c08c880fd6aa58d167055b11c3020f"
Sep 07 11:42:48.574128 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.574103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926912" event={"ID":"f6c70181-c1ba-4088-9b5f-41769425183a","Type":"ContainerStarted","Data":"2710e95eac618589ea4157903e312e855a604f43f3aa30ac22020d27d1598a54"}
Sep 07 11:42:48.584958 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.584936    2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7221-140711668926912" oldPodUID="e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a" podUID="f6c70181-c1ba-4088-9b5f-41769425183a"
Sep 07 11:42:48.960944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:48.960911    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a" path="/var/lib/kubelet/pods/e5d5924c-aff9-4c87-bcc1-5a2d1aa7368a/volumes"
Sep 07 11:42:49.121420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.121389    2578 scope.go:117] "RemoveContainer" containerID="6af62c3753ffa495101b1215408ab187fa2336f484b151592bb6b7bdd9eaf010"
Sep 07 11:42:49.130114 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.130097    2578 scope.go:117] "RemoveContainer" containerID="39137d40f350f23f5c31f3436a82d91d01219f5ed6073e07d9cfee8d7f47e45c"
Sep 07 11:42:49.141566 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.141544    2578 scope.go:117] "RemoveContainer" containerID="c96ab2095ac3b46b38b4cb4ba27c04e39100bd17abd84ab099c0a1bfa7f638ff"
Sep 07 11:42:49.388252 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.388226    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:42:49.393456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.393432    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:42:49.393590 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.393551    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:42:49.398761 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.398745    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:42:49.581045 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.580966    2578 generic.go:356] "Generic (PLEG): container finished" podID="f6c70181-c1ba-4088-9b5f-41769425183a" containerID="7cdf624280ff8702614cca1661a6be851535f92f040241ba8f4ea9ad5a0fbc65" exitCode=0
Sep 07 11:42:49.581045 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:49.581035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7221-140711668926912" event={"ID":"f6c70181-c1ba-4088-9b5f-41769425183a","Type":"ContainerDied","Data":"7cdf624280ff8702614cca1661a6be851535f92f040241ba8f4ea9ad5a0fbc65"}
Sep 07 11:42:50.699994 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.699961    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"]
Sep 07 11:42:50.704404 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.704384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.707070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.707046    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"openshift-service-ca.crt\""
Sep 07 11:42:50.707542 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.707525    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"auth-test-model-0f0a2f-kserve-self-signed-certs\""
Sep 07 11:42:50.707639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.707545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"default-dockercfg-8qzws\""
Sep 07 11:42:50.708037 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.708016    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"kube-root-ca.crt\""
Sep 07 11:42:50.715622 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.715600    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"]
Sep 07 11:42:50.729688 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.729671    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:50.731887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.731856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:50.765054 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.765031    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7221-140711668926912_f6c70181-c1ba-4088-9b5f-41769425183a/test-curl-7221-140711668926912/0.log"
Sep 07 11:42:50.792840 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.792804    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:50.796358 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.796336    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7221-140711668926912"]
Sep 07 11:42:50.808201 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.808180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd47r\" (UniqueName: \"kubernetes.io/projected/f6c70181-c1ba-4088-9b5f-41769425183a-kube-api-access-sd47r\") pod \"f6c70181-c1ba-4088-9b5f-41769425183a\" (UID: \"f6c70181-c1ba-4088-9b5f-41769425183a\") "
Sep 07 11:42:50.808383 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.808367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-home\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.808447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.808390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-model-cache\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.808447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.808412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcp5v\" (UniqueName: \"kubernetes.io/projected/5f206f09-31fd-4bad-9637-28743bbd7a78-kube-api-access-wcp5v\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.808447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.808429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-tmp-dir\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.808625 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.808579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-dshm\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.808625 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.808615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f206f09-31fd-4bad-9637-28743bbd7a78-tls-certs\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.810303 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.810283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c70181-c1ba-4088-9b5f-41769425183a-kube-api-access-sd47r" (OuterVolumeSpecName: "kube-api-access-sd47r") pod "f6c70181-c1ba-4088-9b5f-41769425183a" (UID: "f6c70181-c1ba-4088-9b5f-41769425183a"). InnerVolumeSpecName "kube-api-access-sd47r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:42:50.909865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.909838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-dshm\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.909872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f206f09-31fd-4bad-9637-28743bbd7a78-tls-certs\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.909937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-home\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.909955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-model-cache\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.909977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcp5v\" (UniqueName: \"kubernetes.io/projected/5f206f09-31fd-4bad-9637-28743bbd7a78-kube-api-access-wcp5v\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.910019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-tmp-dir\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.910083    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd47r\" (UniqueName: \"kubernetes.io/projected/f6c70181-c1ba-4088-9b5f-41769425183a-kube-api-access-sd47r\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:42:50.910373 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.910348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-home\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910426 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.910371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-tmp-dir\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.910631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.910609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-model-cache\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.912164 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.912139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-dshm\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.912451 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.912429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f206f09-31fd-4bad-9637-28743bbd7a78-tls-certs\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.919438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.919418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcp5v\" (UniqueName: \"kubernetes.io/projected/5f206f09-31fd-4bad-9637-28743bbd7a78-kube-api-access-wcp5v\") pod \"auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:50.960616 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:50.960592    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c70181-c1ba-4088-9b5f-41769425183a" path="/var/lib/kubelet/pods/f6c70181-c1ba-4088-9b5f-41769425183a/volumes"
Sep 07 11:42:51.035227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.035199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:42:51.310107 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.310077    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"]
Sep 07 11:42:51.311201 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:51.311170    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f206f09_31fd_4bad_9637_28743bbd7a78.slice/crio-569bc8e044c0f8c8b1a9d9d6a0c9f5fa09ebc321dfeece43b54705532293aa49 WatchSource:0}: Error finding container 569bc8e044c0f8c8b1a9d9d6a0c9f5fa09ebc321dfeece43b54705532293aa49: Status 404 returned error can't find the container with id 569bc8e044c0f8c8b1a9d9d6a0c9f5fa09ebc321dfeece43b54705532293aa49
Sep 07 11:42:51.447036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.447000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:51.593147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.593049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn" event={"ID":"5f206f09-31fd-4bad-9637-28743bbd7a78","Type":"ContainerStarted","Data":"32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1"}
Sep 07 11:42:51.593147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.593088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn" event={"ID":"5f206f09-31fd-4bad-9637-28743bbd7a78","Type":"ContainerStarted","Data":"569bc8e044c0f8c8b1a9d9d6a0c9f5fa09ebc321dfeece43b54705532293aa49"}
Sep 07 11:42:51.594381 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.594359    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7221-140711668926912"
Sep 07 11:42:51.594506 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.594403    2578 scope.go:117] "RemoveContainer" containerID="7cdf624280ff8702614cca1661a6be851535f92f040241ba8f4ea9ad5a0fbc65"
Sep 07 11:42:51.615476 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.615424    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn" podStartSLOduration=1.615410097 podStartE2EDuration="1.615410097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:42:51.61274292 +0000 UTC m=+1203.087591090" watchObservedRunningTime="2026-09-07 11:42:51.615410097 +0000 UTC m=+1203.090258247"
Sep 07 11:42:51.761396 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:51.761363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:52.115319 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:52.115274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:54.387336 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:54.387305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:54.487370 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:54.487340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:55.583280 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:55.583250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:55.695864 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:55.695827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:55.848492 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:55.848396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:56.218156 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.218129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:56.316448 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.316416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:42:56.343936 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.343904    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"]
Sep 07 11:42:56.344315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.344302    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c70181-c1ba-4088-9b5f-41769425183a" containerName="test-curl-7221-140711668926912"
Sep 07 11:42:56.344363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.344316    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c70181-c1ba-4088-9b5f-41769425183a" containerName="test-curl-7221-140711668926912"
Sep 07 11:42:56.344399 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.344388    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c70181-c1ba-4088-9b5f-41769425183a" containerName="test-curl-7221-140711668926912"
Sep 07 11:42:56.348756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.348736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.352824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.352801    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8108f8fd-dockercfg-wq2xq\""
Sep 07 11:42:56.353181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.353163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8108f8fd\""
Sep 07 11:42:56.373102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.372417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"]
Sep 07 11:42:56.457401 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.457371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9a3258-3b16-412b-8257-c8818740d67d-plugins-config-volume\") pod \"payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") " pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.457614 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.457457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qv6\" (UniqueName: \"kubernetes.io/projected/6e9a3258-3b16-412b-8257-c8818740d67d-kube-api-access-l2qv6\") pod \"payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") " pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.558770 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.558688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9a3258-3b16-412b-8257-c8818740d67d-plugins-config-volume\") pod \"payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") " pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.558911 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.558789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qv6\" (UniqueName: \"kubernetes.io/projected/6e9a3258-3b16-412b-8257-c8818740d67d-kube-api-access-l2qv6\") pod \"payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") " pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.559495 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.559456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9a3258-3b16-412b-8257-c8818740d67d-plugins-config-volume\") pod \"payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") " pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.569381 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.569350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qv6\" (UniqueName: \"kubernetes.io/projected/6e9a3258-3b16-412b-8257-c8818740d67d-kube-api-access-l2qv6\") pod \"payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") " pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.659757 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.659729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:56.942204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:56.942171    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"]
Sep 07 11:42:56.943991 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:42:56.943965    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9a3258_3b16_412b_8257_c8818740d67d.slice/crio-a9edd31405c9f4155d122694ec2e737b3d9655a91c86b0170b323ab7cc494910 WatchSource:0}: Error finding container a9edd31405c9f4155d122694ec2e737b3d9655a91c86b0170b323ab7cc494910: Status 404 returned error can't find the container with id a9edd31405c9f4155d122694ec2e737b3d9655a91c86b0170b323ab7cc494910
Sep 07 11:42:57.632990 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:57.632958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44" event={"ID":"6e9a3258-3b16-412b-8257-c8818740d67d","Type":"ContainerStarted","Data":"4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9"}
Sep 07 11:42:57.633154 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:57.633011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44" event={"ID":"6e9a3258-3b16-412b-8257-c8818740d67d","Type":"ContainerStarted","Data":"a9edd31405c9f4155d122694ec2e737b3d9655a91c86b0170b323ab7cc494910"}
Sep 07 11:42:57.633205 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:57.633148    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:42:57.657836 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:57.657783    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44" podStartSLOduration=1.6577609880000002 podStartE2EDuration="1.657760988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:42:57.657011357 +0000 UTC m=+1209.131859529" watchObservedRunningTime="2026-09-07 11:42:57.657760988 +0000 UTC m=+1209.132609174"
Sep 07 11:42:59.300025 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:42:59.299990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:43:01.035580 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:01.035536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:43:01.035997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:01.035592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:43:01.048097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:01.048066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:43:01.667571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:01.667540    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:43:08.641654 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:08.641625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:43:09.255850 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:09.255815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:43:09.355631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:09.355603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:43:12.493830 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:12.493794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:43:19.434741 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:19.434695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:43:27.895601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:27.895558    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:43:32.439592 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:32.439557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:43:44.993998 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:44.993910    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:43:46.393517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.393482    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"]
Sep 07 11:43:46.398446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.398422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.402187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.402151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8108f8fd\"/\"default-dockercfg-cghfw\""
Sep 07 11:43:46.402308 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.402196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8108f8fd\"/\"openshift-service-ca.crt\""
Sep 07 11:43:46.402377 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.402333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8108f8fd\"/\"e2e-lifecycle-model-8108f8fd-kserve-self-signed-certs\""
Sep 07 11:43:46.402488 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.402407    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8108f8fd\"/\"kube-root-ca.crt\""
Sep 07 11:43:46.405439 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.405417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"]
Sep 07 11:43:46.485573 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.485545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-tmp-dir\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.485711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.485585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8ebff3-be57-405e-9744-74ba1dabaf56-tls-certs\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.485711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.485696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbrb\" (UniqueName: \"kubernetes.io/projected/6d8ebff3-be57-405e-9744-74ba1dabaf56-kube-api-access-vrbrb\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.485795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.485727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-home\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.485795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.485757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-model-cache\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.485795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.485784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-dshm\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.490073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.490048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:43:46.587263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbrb\" (UniqueName: \"kubernetes.io/projected/6d8ebff3-be57-405e-9744-74ba1dabaf56-kube-api-access-vrbrb\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-home\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-model-cache\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-dshm\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-tmp-dir\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8ebff3-be57-405e-9744-74ba1dabaf56-tls-certs\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-home\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587763 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-model-cache\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.587827 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.587805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-tmp-dir\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.589670 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.589649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-dshm\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.589794 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.589776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8ebff3-be57-405e-9744-74ba1dabaf56-tls-certs\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.596449 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.596426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbrb\" (UniqueName: \"kubernetes.io/projected/6d8ebff3-be57-405e-9744-74ba1dabaf56-kube-api-access-vrbrb\") pod \"e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") " pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.709778 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.709708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:46.991119 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:46.991094    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"]
Sep 07 11:43:46.992805 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:43:46.992773    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8ebff3_be57_405e_9744_74ba1dabaf56.slice/crio-da7e1f9ce979f51673b8bfb5107ca88b4ab0cb3bfd299c7be264e2c2242e1594 WatchSource:0}: Error finding container da7e1f9ce979f51673b8bfb5107ca88b4ab0cb3bfd299c7be264e2c2242e1594: Status 404 returned error can't find the container with id da7e1f9ce979f51673b8bfb5107ca88b4ab0cb3bfd299c7be264e2c2242e1594
Sep 07 11:43:47.934682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:47.934641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7" event={"ID":"6d8ebff3-be57-405e-9744-74ba1dabaf56","Type":"ContainerStarted","Data":"96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c"}
Sep 07 11:43:47.934682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:47.934689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7" event={"ID":"6d8ebff3-be57-405e-9744-74ba1dabaf56","Type":"ContainerStarted","Data":"da7e1f9ce979f51673b8bfb5107ca88b4ab0cb3bfd299c7be264e2c2242e1594"}
Sep 07 11:43:47.954890 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:47.954829    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7" podStartSLOduration=1.95480906 podStartE2EDuration="1.95480906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:43:47.954181059 +0000 UTC m=+1259.429029228" watchObservedRunningTime="2026-09-07 11:43:47.95480906 +0000 UTC m=+1259.429657210"
Sep 07 11:43:49.174995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:49.174963    2578 scope.go:117] "RemoveContainer" containerID="07c95552a4123db5c301bce1c04e345290fc73f88e609508a5c6bf1977e47b0e"
Sep 07 11:43:56.710867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:56.710826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:56.710867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:56.710877    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:56.723179 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:56.723154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:43:56.998205 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:43:56.998128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:44:04.408235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:04.408198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:44:21.303957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:21.303924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:44:25.034286 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.034241    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"]
Sep 07 11:44:25.037936 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.037918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"
Sep 07 11:44:25.040553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.040522    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:44:25.040697 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.040631    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:44:25.047131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.047109    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"]
Sep 07 11:44:25.114822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.114785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69287038-6e26-4ed8-99cd-ef544dc6fdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb\" (UID: \"69287038-6e26-4ed8-99cd-ef544dc6fdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"
Sep 07 11:44:25.152295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.152242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:44:25.216108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.216068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69287038-6e26-4ed8-99cd-ef544dc6fdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb\" (UID: \"69287038-6e26-4ed8-99cd-ef544dc6fdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"
Sep 07 11:44:25.216796 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.216769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69287038-6e26-4ed8-99cd-ef544dc6fdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb\" (UID: \"69287038-6e26-4ed8-99cd-ef544dc6fdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"
Sep 07 11:44:25.350171 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.350064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"
Sep 07 11:44:25.621385 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:25.621353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"]
Sep 07 11:44:25.623340 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:44:25.623308    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69287038_6e26_4ed8_99cd_ef544dc6fdc4.slice/crio-fded0d0c33539b485a852285aebe8655c11966311945767d1f5b30f1b0fb8225 WatchSource:0}: Error finding container fded0d0c33539b485a852285aebe8655c11966311945767d1f5b30f1b0fb8225: Status 404 returned error can't find the container with id fded0d0c33539b485a852285aebe8655c11966311945767d1f5b30f1b0fb8225
Sep 07 11:44:26.163861 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:26.163762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb" event={"ID":"69287038-6e26-4ed8-99cd-ef544dc6fdc4","Type":"ContainerStarted","Data":"68ac0c559e53a31ba2663b0fdaeaeff0dce7b66ff6f3e5ccfa290dc9894cd16f"}
Sep 07 11:44:26.163861 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:26.163825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb" event={"ID":"69287038-6e26-4ed8-99cd-ef544dc6fdc4","Type":"ContainerStarted","Data":"fded0d0c33539b485a852285aebe8655c11966311945767d1f5b30f1b0fb8225"}
Sep 07 11:44:26.179785 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:26.179724    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb" podStartSLOduration=1.179702048 podStartE2EDuration="1.179702048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:44:26.178978996 +0000 UTC m=+1297.653827168" watchObservedRunningTime="2026-09-07 11:44:26.179702048 +0000 UTC m=+1297.654550197"
Sep 07 11:44:27.171856 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:27.171818    2578 generic.go:356] "Generic (PLEG): container finished" podID="69287038-6e26-4ed8-99cd-ef544dc6fdc4" containerID="68ac0c559e53a31ba2663b0fdaeaeff0dce7b66ff6f3e5ccfa290dc9894cd16f" exitCode=0
Sep 07 11:44:27.172349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:27.171904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb" event={"ID":"69287038-6e26-4ed8-99cd-ef544dc6fdc4","Type":"ContainerDied","Data":"68ac0c559e53a31ba2663b0fdaeaeff0dce7b66ff6f3e5ccfa290dc9894cd16f"}
Sep 07 11:44:28.316831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:28.316805    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"
Sep 07 11:44:28.345108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:28.345078    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69287038-6e26-4ed8-99cd-ef544dc6fdc4-maas-api-service-ca\") pod \"69287038-6e26-4ed8-99cd-ef544dc6fdc4\" (UID: \"69287038-6e26-4ed8-99cd-ef544dc6fdc4\") "
Sep 07 11:44:28.345434 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:28.345410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69287038-6e26-4ed8-99cd-ef544dc6fdc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69287038-6e26-4ed8-99cd-ef544dc6fdc4" (UID: "69287038-6e26-4ed8-99cd-ef544dc6fdc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:44:28.345597 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:28.345577    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69287038-6e26-4ed8-99cd-ef544dc6fdc4-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:44:29.184854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:29.184828    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"
Sep 07 11:44:29.185010 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:29.184829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb" event={"ID":"69287038-6e26-4ed8-99cd-ef544dc6fdc4","Type":"ContainerDied","Data":"fded0d0c33539b485a852285aebe8655c11966311945767d1f5b30f1b0fb8225"}
Sep 07 11:44:29.185010 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:29.184942    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fded0d0c33539b485a852285aebe8655c11966311945767d1f5b30f1b0fb8225"
Sep 07 11:44:39.397031 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:39.396991    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:44:40.512294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.512261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:44:40.539051 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.539019    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"]
Sep 07 11:44:40.539280 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.539256    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44" podUID="6e9a3258-3b16-412b-8257-c8818740d67d" containerName="payload-processing" containerID="cri-o://4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9" gracePeriod=30
Sep 07 11:44:40.612778 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.612747    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:44:40.715963 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.715929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:44:40.719058 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.719033    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"]
Sep 07 11:44:40.722906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.722876    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8108f8fd-vrdtb"]
Sep 07 11:44:40.798915 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.798882    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:44:40.813980 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.813938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:44:40.861382 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.861350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2qv6\" (UniqueName: \"kubernetes.io/projected/6e9a3258-3b16-412b-8257-c8818740d67d-kube-api-access-l2qv6\") pod \"6e9a3258-3b16-412b-8257-c8818740d67d\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") "
Sep 07 11:44:40.861548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.861451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9a3258-3b16-412b-8257-c8818740d67d-plugins-config-volume\") pod \"6e9a3258-3b16-412b-8257-c8818740d67d\" (UID: \"6e9a3258-3b16-412b-8257-c8818740d67d\") "
Sep 07 11:44:40.861862 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.861837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9a3258-3b16-412b-8257-c8818740d67d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e9a3258-3b16-412b-8257-c8818740d67d" (UID: "6e9a3258-3b16-412b-8257-c8818740d67d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:44:40.863500 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.863449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9a3258-3b16-412b-8257-c8818740d67d-kube-api-access-l2qv6" (OuterVolumeSpecName: "kube-api-access-l2qv6") pod "6e9a3258-3b16-412b-8257-c8818740d67d" (UID: "6e9a3258-3b16-412b-8257-c8818740d67d"). InnerVolumeSpecName "kube-api-access-l2qv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:44:40.960438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.960408    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69287038-6e26-4ed8-99cd-ef544dc6fdc4" path="/var/lib/kubelet/pods/69287038-6e26-4ed8-99cd-ef544dc6fdc4/volumes"
Sep 07 11:44:40.962221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.962204    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qv6\" (UniqueName: \"kubernetes.io/projected/6e9a3258-3b16-412b-8257-c8818740d67d-kube-api-access-l2qv6\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:44:40.962275 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:40.962224    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9a3258-3b16-412b-8257-c8818740d67d-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:44:41.259714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.259679    2578 generic.go:356] "Generic (PLEG): container finished" podID="6e9a3258-3b16-412b-8257-c8818740d67d" containerID="4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9" exitCode=0
Sep 07 11:44:41.259886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.259726    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"
Sep 07 11:44:41.259886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.259765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44" event={"ID":"6e9a3258-3b16-412b-8257-c8818740d67d","Type":"ContainerDied","Data":"4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9"}
Sep 07 11:44:41.259886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.259822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44" event={"ID":"6e9a3258-3b16-412b-8257-c8818740d67d","Type":"ContainerDied","Data":"a9edd31405c9f4155d122694ec2e737b3d9655a91c86b0170b323ab7cc494910"}
Sep 07 11:44:41.259886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.259845    2578 scope.go:117] "RemoveContainer" containerID="4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9"
Sep 07 11:44:41.269617 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.269597    2578 scope.go:117] "RemoveContainer" containerID="4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9"
Sep 07 11:44:41.269857 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:44:41.269835    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9\": container with ID starting with 4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9 not found: ID does not exist" containerID="4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9"
Sep 07 11:44:41.269935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.269866    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9"} err="failed to get container status \"4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9\": rpc error: code = NotFound desc = could not find container \"4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9\": container with ID starting with 4ddf0ce4bbfed5ea6a307f860684e9e6f6c30b3d22518a8dbe920322a23f9ba9 not found: ID does not exist"
Sep 07 11:44:41.279549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.279453    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"]
Sep 07 11:44:41.283413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:41.283389    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8108f8fd-87dddb857-8dv44"]
Sep 07 11:44:42.960824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:42.960792    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e9a3258-3b16-412b-8257-c8818740d67d" path="/var/lib/kubelet/pods/6e9a3258-3b16-412b-8257-c8818740d67d/volumes"
Sep 07 11:44:44.070278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:44.070245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:44:49.050772 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:49.050735    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"]
Sep 07 11:44:49.147761 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:49.147725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:44:49.201623 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:49.201599    2578 scope.go:117] "RemoveContainer" containerID="208e937628bb455f46bea7a935d6eb340d2f805a178c307e33afde2d5eee8ca1"
Sep 07 11:44:49.210258 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:49.210239    2578 scope.go:117] "RemoveContainer" containerID="38db1fadd657bebdf82bb2e1d61dd5f479433399f0d3c22158a5334926a86008"
Sep 07 11:44:59.422931 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:44:59.422894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:45:00.145913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.145876    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg"]
Sep 07 11:45:00.146285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.146272    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e9a3258-3b16-412b-8257-c8818740d67d" containerName="payload-processing"
Sep 07 11:45:00.146332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.146287    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e9a3258-3b16-412b-8257-c8818740d67d" containerName="payload-processing"
Sep 07 11:45:00.146332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.146302    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69287038-6e26-4ed8-99cd-ef544dc6fdc4" containerName="revoke-keys"
Sep 07 11:45:00.146332 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.146308    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69287038-6e26-4ed8-99cd-ef544dc6fdc4" containerName="revoke-keys"
Sep 07 11:45:00.146428 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.146371    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e9a3258-3b16-412b-8257-c8818740d67d" containerName="payload-processing"
Sep 07 11:45:00.146428 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.146382    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="69287038-6e26-4ed8-99cd-ef544dc6fdc4" containerName="revoke-keys"
Sep 07 11:45:00.150827 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.150803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg"
Sep 07 11:45:00.151928 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.151908    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"]
Sep 07 11:45:00.160301 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.160274    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:45:00.160892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.160432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"
Sep 07 11:45:00.160892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.160347    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"]
Sep 07 11:45:00.171447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.171246    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg"]
Sep 07 11:45:00.171447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.171346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"
Sep 07 11:45:00.171623 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.171512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"]
Sep 07 11:45:00.172102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.172084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg"
Sep 07 11:45:00.178685 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.178643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"]
Sep 07 11:45:00.185068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.185048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"
Sep 07 11:45:00.225494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.225161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"
Sep 07 11:45:00.459951 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.459922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg"]
Sep 07 11:45:00.461116 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:45:00.461092    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aafe403_36ae_4bbd_9f77_dc0c506e71cf.slice/crio-9882ce1078bbb17344132957b634e0271fcfc413ba1af860bbd1e4ed1b1642d9 WatchSource:0}: Error finding container 9882ce1078bbb17344132957b634e0271fcfc413ba1af860bbd1e4ed1b1642d9: Status 404 returned error can't find the container with id 9882ce1078bbb17344132957b634e0271fcfc413ba1af860bbd1e4ed1b1642d9
Sep 07 11:45:00.488640 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.488570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"]
Sep 07 11:45:00.498309 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:45:00.498281    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a0a941_54a7_49c8_9ae3_101effc73dbf.slice/crio-04e5c9c5ff41dc0701688e55a940f3290555e401e26c8ec600ff09a91c213101 WatchSource:0}: Error finding container 04e5c9c5ff41dc0701688e55a940f3290555e401e26c8ec600ff09a91c213101: Status 404 returned error can't find the container with id 04e5c9c5ff41dc0701688e55a940f3290555e401e26c8ec600ff09a91c213101
Sep 07 11:45:00.528813 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:00.528781    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"]
Sep 07 11:45:00.533694 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:45:00.533666    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3bdcf7_5b17_490e_a904_c287a76a236a.slice/crio-fc6a82f4ba75a57a5aad5ead4267bcf3acfe4f183518f170f1a998caa01f8720 WatchSource:0}: Error finding container fc6a82f4ba75a57a5aad5ead4267bcf3acfe4f183518f170f1a998caa01f8720: Status 404 returned error can't find the container with id fc6a82f4ba75a57a5aad5ead4267bcf3acfe4f183518f170f1a998caa01f8720
Sep 07 11:45:01.382033 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.381997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn" event={"ID":"bd3bdcf7-5b17-490e-a904-c287a76a236a","Type":"ContainerStarted","Data":"8ed91dad034a51c28f5b6abb09545f36279a5ecebe3f3d529d1f9fa2bb5770f5"}
Sep 07 11:45:01.382135 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.382068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn" event={"ID":"bd3bdcf7-5b17-490e-a904-c287a76a236a","Type":"ContainerStarted","Data":"fc6a82f4ba75a57a5aad5ead4267bcf3acfe4f183518f170f1a998caa01f8720"}
Sep 07 11:45:01.383603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.383580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf" event={"ID":"30a0a941-54a7-49c8-9ae3-101effc73dbf","Type":"ContainerStarted","Data":"0a67a5d9c18ebb6efb69976fa7f4f4475a0b2dbc07150fbb96ea636e95dbc383"}
Sep 07 11:45:01.383713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.383617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf" event={"ID":"30a0a941-54a7-49c8-9ae3-101effc73dbf","Type":"ContainerStarted","Data":"04e5c9c5ff41dc0701688e55a940f3290555e401e26c8ec600ff09a91c213101"}
Sep 07 11:45:01.384953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.384934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg" event={"ID":"9aafe403-36ae-4bbd-9f77-dc0c506e71cf","Type":"ContainerStarted","Data":"6858cd1662a6f0871b1a4b7576ca33bf92179e2df6d096e2bae5bebf4f342e68"}
Sep 07 11:45:01.385035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.384969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg" event={"ID":"9aafe403-36ae-4bbd-9f77-dc0c506e71cf","Type":"ContainerStarted","Data":"9882ce1078bbb17344132957b634e0271fcfc413ba1af860bbd1e4ed1b1642d9"}
Sep 07 11:45:01.399787 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.399749    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn" podStartSLOduration=1.399734789 podStartE2EDuration="1.399734789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:45:01.3990531 +0000 UTC m=+1332.873901266" watchObservedRunningTime="2026-09-07 11:45:01.399734789 +0000 UTC m=+1332.874582941"
Sep 07 11:45:01.438148 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:01.438093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf" podStartSLOduration=1.438078263 podStartE2EDuration="1.438078263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:45:01.437965157 +0000 UTC m=+1332.912813328" watchObservedRunningTime="2026-09-07 11:45:01.438078263 +0000 UTC m=+1332.912926415"
Sep 07 11:45:02.394008 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.393968    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd3bdcf7-5b17-490e-a904-c287a76a236a" containerID="8ed91dad034a51c28f5b6abb09545f36279a5ecebe3f3d529d1f9fa2bb5770f5" exitCode=0
Sep 07 11:45:02.394454 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.394053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn" event={"ID":"bd3bdcf7-5b17-490e-a904-c287a76a236a","Type":"ContainerDied","Data":"8ed91dad034a51c28f5b6abb09545f36279a5ecebe3f3d529d1f9fa2bb5770f5"}
Sep 07 11:45:02.395617 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.395593    2578 generic.go:356] "Generic (PLEG): container finished" podID="30a0a941-54a7-49c8-9ae3-101effc73dbf" containerID="0a67a5d9c18ebb6efb69976fa7f4f4475a0b2dbc07150fbb96ea636e95dbc383" exitCode=0
Sep 07 11:45:02.395725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.395676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf" event={"ID":"30a0a941-54a7-49c8-9ae3-101effc73dbf","Type":"ContainerDied","Data":"0a67a5d9c18ebb6efb69976fa7f4f4475a0b2dbc07150fbb96ea636e95dbc383"}
Sep 07 11:45:02.397082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.397062    2578 generic.go:356] "Generic (PLEG): container finished" podID="9aafe403-36ae-4bbd-9f77-dc0c506e71cf" containerID="6858cd1662a6f0871b1a4b7576ca33bf92179e2df6d096e2bae5bebf4f342e68" exitCode=0
Sep 07 11:45:02.397169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.397086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg" event={"ID":"9aafe403-36ae-4bbd-9f77-dc0c506e71cf","Type":"ContainerDied","Data":"6858cd1662a6f0871b1a4b7576ca33bf92179e2df6d096e2bae5bebf4f342e68"}
Sep 07 11:45:02.536549 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.536517    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7224-062991"]
Sep 07 11:45:02.540864 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.540843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:02.543338 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.543313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhmxq\""
Sep 07 11:45:02.546322 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.546291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7224-062991"]
Sep 07 11:45:02.551139 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.551119    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg"
Sep 07 11:45:02.555804 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.555778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vjn\" (UniqueName: \"kubernetes.io/projected/67c82285-dbfd-453b-aea0-fdbfe519923b-kube-api-access-f7vjn\") pod \"mt-curl-7224-062991\" (UID: \"67c82285-dbfd-453b-aea0-fdbfe519923b\") " pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:02.656899 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.656812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vjn\" (UniqueName: \"kubernetes.io/projected/67c82285-dbfd-453b-aea0-fdbfe519923b-kube-api-access-f7vjn\") pod \"mt-curl-7224-062991\" (UID: \"67c82285-dbfd-453b-aea0-fdbfe519923b\") " pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:02.665999 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.665975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vjn\" (UniqueName: \"kubernetes.io/projected/67c82285-dbfd-453b-aea0-fdbfe519923b-kube-api-access-f7vjn\") pod \"mt-curl-7224-062991\" (UID: \"67c82285-dbfd-453b-aea0-fdbfe519923b\") " pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:02.856970 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:02.856936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:03.129701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:03.129671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7224-062991"]
Sep 07 11:45:03.130846 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:45:03.130824    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67c82285_dbfd_453b_aea0_fdbfe519923b.slice/crio-006a0f74909db9f5507068dad759334db1986d4309412e87c55f29dd31e8f2df WatchSource:0}: Error finding container 006a0f74909db9f5507068dad759334db1986d4309412e87c55f29dd31e8f2df: Status 404 returned error can't find the container with id 006a0f74909db9f5507068dad759334db1986d4309412e87c55f29dd31e8f2df
Sep 07 11:45:03.404723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:03.404694    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg"
Sep 07 11:45:03.405126 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:03.404727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813025-wjrbg" event={"ID":"9aafe403-36ae-4bbd-9f77-dc0c506e71cf","Type":"ContainerDied","Data":"9882ce1078bbb17344132957b634e0271fcfc413ba1af860bbd1e4ed1b1642d9"}
Sep 07 11:45:03.405126 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:03.404778    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9882ce1078bbb17344132957b634e0271fcfc413ba1af860bbd1e4ed1b1642d9"
Sep 07 11:45:03.406107 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:03.406078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7224-062991" event={"ID":"67c82285-dbfd-453b-aea0-fdbfe519923b","Type":"ContainerStarted","Data":"006a0f74909db9f5507068dad759334db1986d4309412e87c55f29dd31e8f2df"}
Sep 07 11:45:03.587626 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:03.587599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"
Sep 07 11:45:03.590833 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:03.590813    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"
Sep 07 11:45:04.071186 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.071141    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7" podUID="6d8ebff3-be57-405e-9744-74ba1dabaf56" containerName="main" containerID="cri-o://96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c" gracePeriod=45
Sep 07 11:45:04.227571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.227539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:04.286357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.286325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:04.308301 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.308276    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:45:04.373504 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-dshm\") pod \"6d8ebff3-be57-405e-9744-74ba1dabaf56\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") "
Sep 07 11:45:04.373660 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrbrb\" (UniqueName: \"kubernetes.io/projected/6d8ebff3-be57-405e-9744-74ba1dabaf56-kube-api-access-vrbrb\") pod \"6d8ebff3-be57-405e-9744-74ba1dabaf56\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") "
Sep 07 11:45:04.373660 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373571    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-tmp-dir\") pod \"6d8ebff3-be57-405e-9744-74ba1dabaf56\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") "
Sep 07 11:45:04.373739 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373665    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-model-cache\") pod \"6d8ebff3-be57-405e-9744-74ba1dabaf56\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") "
Sep 07 11:45:04.373739 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-home\") pod \"6d8ebff3-be57-405e-9744-74ba1dabaf56\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") "
Sep 07 11:45:04.373853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8ebff3-be57-405e-9744-74ba1dabaf56-tls-certs\") pod \"6d8ebff3-be57-405e-9744-74ba1dabaf56\" (UID: \"6d8ebff3-be57-405e-9744-74ba1dabaf56\") "
Sep 07 11:45:04.374066 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d8ebff3-be57-405e-9744-74ba1dabaf56" (UID: "6d8ebff3-be57-405e-9744-74ba1dabaf56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:04.374066 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.373962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d8ebff3-be57-405e-9744-74ba1dabaf56" (UID: "6d8ebff3-be57-405e-9744-74ba1dabaf56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:04.374066 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.374031    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-home" (OuterVolumeSpecName: "home") pod "6d8ebff3-be57-405e-9744-74ba1dabaf56" (UID: "6d8ebff3-be57-405e-9744-74ba1dabaf56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:04.374273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.374114    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:04.374273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.374133    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:04.374273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.374147    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:04.375738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.375712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8ebff3-be57-405e-9744-74ba1dabaf56-kube-api-access-vrbrb" (OuterVolumeSpecName: "kube-api-access-vrbrb") pod "6d8ebff3-be57-405e-9744-74ba1dabaf56" (UID: "6d8ebff3-be57-405e-9744-74ba1dabaf56"). InnerVolumeSpecName "kube-api-access-vrbrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:45:04.376016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.375773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8ebff3-be57-405e-9744-74ba1dabaf56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d8ebff3-be57-405e-9744-74ba1dabaf56" (UID: "6d8ebff3-be57-405e-9744-74ba1dabaf56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:45:04.376016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.375991    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-dshm" (OuterVolumeSpecName: "dshm") pod "6d8ebff3-be57-405e-9744-74ba1dabaf56" (UID: "6d8ebff3-be57-405e-9744-74ba1dabaf56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:04.414437 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.414410    2578 generic.go:356] "Generic (PLEG): container finished" podID="6d8ebff3-be57-405e-9744-74ba1dabaf56" containerID="96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c" exitCode=0
Sep 07 11:45:04.414821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.414454    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"
Sep 07 11:45:04.414821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.414501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7" event={"ID":"6d8ebff3-be57-405e-9744-74ba1dabaf56","Type":"ContainerDied","Data":"96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c"}
Sep 07 11:45:04.414821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.414559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7" event={"ID":"6d8ebff3-be57-405e-9744-74ba1dabaf56","Type":"ContainerDied","Data":"da7e1f9ce979f51673b8bfb5107ca88b4ab0cb3bfd299c7be264e2c2242e1594"}
Sep 07 11:45:04.414821 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.414585    2578 scope.go:117] "RemoveContainer" containerID="96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c"
Sep 07 11:45:04.416315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.416289    2578 generic.go:356] "Generic (PLEG): container finished" podID="67c82285-dbfd-453b-aea0-fdbfe519923b" containerID="6c6cff6f297c37558f7d7115affc6790f3004e42b477ec1d74db96c3ee58262c" exitCode=0
Sep 07 11:45:04.416431 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.416373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7224-062991" event={"ID":"67c82285-dbfd-453b-aea0-fdbfe519923b","Type":"ContainerDied","Data":"6c6cff6f297c37558f7d7115affc6790f3004e42b477ec1d74db96c3ee58262c"}
Sep 07 11:45:04.417687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.417666    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"
Sep 07 11:45:04.417779 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.417688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn" event={"ID":"bd3bdcf7-5b17-490e-a904-c287a76a236a","Type":"ContainerDied","Data":"fc6a82f4ba75a57a5aad5ead4267bcf3acfe4f183518f170f1a998caa01f8720"}
Sep 07 11:45:04.417779 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.417733    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6a82f4ba75a57a5aad5ead4267bcf3acfe4f183518f170f1a998caa01f8720"
Sep 07 11:45:04.419227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.419210    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"
Sep 07 11:45:04.419227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.419220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf" event={"ID":"30a0a941-54a7-49c8-9ae3-101effc73dbf","Type":"ContainerDied","Data":"04e5c9c5ff41dc0701688e55a940f3290555e401e26c8ec600ff09a91c213101"}
Sep 07 11:45:04.419403 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.419260    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04e5c9c5ff41dc0701688e55a940f3290555e401e26c8ec600ff09a91c213101"
Sep 07 11:45:04.426659 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.426643    2578 scope.go:117] "RemoveContainer" containerID="96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c"
Sep 07 11:45:04.426945 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:45:04.426925    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c\": container with ID starting with 96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c not found: ID does not exist" containerID="96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c"
Sep 07 11:45:04.427023 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.426951    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c"} err="failed to get container status \"96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c\": rpc error: code = NotFound desc = could not find container \"96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c\": container with ID starting with 96d63ed622253ea1b9506685a8a3e36b99b98a9efad8114a8a66386ad60bfa6c not found: ID does not exist"
Sep 07 11:45:04.455992 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.455926    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"]
Sep 07 11:45:04.459353 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.459330    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8108f8fd/e2e-lifecycle-model-8108f8fd-kserve-5bdc4ffc8c-9g7r7"]
Sep 07 11:45:04.475605 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.475576    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8ebff3-be57-405e-9744-74ba1dabaf56-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:04.475605 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.475600    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8ebff3-be57-405e-9744-74ba1dabaf56-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:04.475717 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.475613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbrb\" (UniqueName: \"kubernetes.io/projected/6d8ebff3-be57-405e-9744-74ba1dabaf56-kube-api-access-vrbrb\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:04.960788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:04.960753    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8ebff3-be57-405e-9744-74ba1dabaf56" path="/var/lib/kubelet/pods/6d8ebff3-be57-405e-9744-74ba1dabaf56/volumes"
Sep 07 11:45:05.567349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:05.567318    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:05.586225 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:05.586195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7vjn\" (UniqueName: \"kubernetes.io/projected/67c82285-dbfd-453b-aea0-fdbfe519923b-kube-api-access-f7vjn\") pod \"67c82285-dbfd-453b-aea0-fdbfe519923b\" (UID: \"67c82285-dbfd-453b-aea0-fdbfe519923b\") "
Sep 07 11:45:05.587614 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:05.587588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7224-062991_67c82285-dbfd-453b-aea0-fdbfe519923b/mt-curl-7224-062991/0.log"
Sep 07 11:45:05.588446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:05.588415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c82285-dbfd-453b-aea0-fdbfe519923b-kube-api-access-f7vjn" (OuterVolumeSpecName: "kube-api-access-f7vjn") pod "67c82285-dbfd-453b-aea0-fdbfe519923b" (UID: "67c82285-dbfd-453b-aea0-fdbfe519923b"). InnerVolumeSpecName "kube-api-access-f7vjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:45:05.613130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:05.613106    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7224-062991"]
Sep 07 11:45:05.615548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:05.615523    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7224-062991"]
Sep 07 11:45:05.687315 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:05.687258    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vjn\" (UniqueName: \"kubernetes.io/projected/67c82285-dbfd-453b-aea0-fdbfe519923b-kube-api-access-f7vjn\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:06.443923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:06.443895    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006a0f74909db9f5507068dad759334db1986d4309412e87c55f29dd31e8f2df"
Sep 07 11:45:06.443923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:06.443900    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:06.453852 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:45:06.453824    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7224-062991\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="67c82285-dbfd-453b-aea0-fdbfe519923b" pod="opendatahub/mt-curl-7224-062991"
Sep 07 11:45:06.961273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:06.961236    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67c82285-dbfd-453b-aea0-fdbfe519923b" path="/var/lib/kubelet/pods/67c82285-dbfd-453b-aea0-fdbfe519923b/volumes"
Sep 07 11:45:06.992090 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:06.992063    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"]
Sep 07 11:45:07.037393 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:07.037364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:08.098387 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:08.098353    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"]
Sep 07 11:45:08.119035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:08.119000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:11.699873 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:11.699799    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:45:16.522713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:16.522674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:22.011361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.011314    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn" podUID="6b3e2cdd-1090-43c3-bef4-d74b409e3279" containerName="main" containerID="cri-o://13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6" gracePeriod=45
Sep 07 11:45:22.249852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.249825    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:45:22.324402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-dshm\") pod \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") "
Sep 07 11:45:22.324570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-model-cache\") pod \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") "
Sep 07 11:45:22.324570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324431    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tmp-dir\") pod \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") "
Sep 07 11:45:22.324570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-home\") pod \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") "
Sep 07 11:45:22.324570 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324549    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr7pk\" (UniqueName: \"kubernetes.io/projected/6b3e2cdd-1090-43c3-bef4-d74b409e3279-kube-api-access-sr7pk\") pod \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") "
Sep 07 11:45:22.324855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324599    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tls-certs\") pod \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\" (UID: \"6b3e2cdd-1090-43c3-bef4-d74b409e3279\") "
Sep 07 11:45:22.324855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324702    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b3e2cdd-1090-43c3-bef4-d74b409e3279" (UID: "6b3e2cdd-1090-43c3-bef4-d74b409e3279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:22.324855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324734    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b3e2cdd-1090-43c3-bef4-d74b409e3279" (UID: "6b3e2cdd-1090-43c3-bef4-d74b409e3279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:22.324855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-home" (OuterVolumeSpecName: "home") pod "6b3e2cdd-1090-43c3-bef4-d74b409e3279" (UID: "6b3e2cdd-1090-43c3-bef4-d74b409e3279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:22.325007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324930    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:22.325007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324943    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:22.325007 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.324951    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:22.326586 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.326563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-dshm" (OuterVolumeSpecName: "dshm") pod "6b3e2cdd-1090-43c3-bef4-d74b409e3279" (UID: "6b3e2cdd-1090-43c3-bef4-d74b409e3279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:22.326673 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.326638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b3e2cdd-1090-43c3-bef4-d74b409e3279" (UID: "6b3e2cdd-1090-43c3-bef4-d74b409e3279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:45:22.326720 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.326665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3e2cdd-1090-43c3-bef4-d74b409e3279-kube-api-access-sr7pk" (OuterVolumeSpecName: "kube-api-access-sr7pk") pod "6b3e2cdd-1090-43c3-bef4-d74b409e3279" (UID: "6b3e2cdd-1090-43c3-bef4-d74b409e3279"). InnerVolumeSpecName "kube-api-access-sr7pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:45:22.425738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.425713    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3e2cdd-1090-43c3-bef4-d74b409e3279-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:22.425738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.425734    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr7pk\" (UniqueName: \"kubernetes.io/projected/6b3e2cdd-1090-43c3-bef4-d74b409e3279-kube-api-access-sr7pk\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:22.425887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.425743    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3e2cdd-1090-43c3-bef4-d74b409e3279-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:22.550775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.550743    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b3e2cdd-1090-43c3-bef4-d74b409e3279" containerID="13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6" exitCode=0
Sep 07 11:45:22.550886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.550795    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"
Sep 07 11:45:22.550886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.550830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn" event={"ID":"6b3e2cdd-1090-43c3-bef4-d74b409e3279","Type":"ContainerDied","Data":"13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6"}
Sep 07 11:45:22.550886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.550882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn" event={"ID":"6b3e2cdd-1090-43c3-bef4-d74b409e3279","Type":"ContainerDied","Data":"1fa2fead157d8c9e4a2d2896b19fc9458d992240f1072398d3af5537a09f1547"}
Sep 07 11:45:22.551068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.550906    2578 scope.go:117] "RemoveContainer" containerID="13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6"
Sep 07 11:45:22.561116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.561099    2578 scope.go:117] "RemoveContainer" containerID="13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6"
Sep 07 11:45:22.561371 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:45:22.561353    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6\": container with ID starting with 13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6 not found: ID does not exist" containerID="13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6"
Sep 07 11:45:22.561419 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.561377    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6"} err="failed to get container status \"13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6\": rpc error: code = NotFound desc = could not find container \"13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6\": container with ID starting with 13571df2c5ea221edfaedc082aeb084d8137866a7108f52ae2b54fed2d1e84b6 not found: ID does not exist"
Sep 07 11:45:22.574641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.574591    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"]
Sep 07 11:45:22.577854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.577834    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/auth-test-model-6b944b-kserve-666cb6f647-h7kkn"]
Sep 07 11:45:22.960169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:22.960139    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3e2cdd-1090-43c3-bef4-d74b409e3279" path="/var/lib/kubelet/pods/6b3e2cdd-1090-43c3-bef4-d74b409e3279/volumes"
Sep 07 11:45:23.119100 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.119051    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn" podUID="5f206f09-31fd-4bad-9637-28743bbd7a78" containerName="main" containerID="cri-o://32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1" gracePeriod=45
Sep 07 11:45:23.373788 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.373760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:45:23.431700 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.431672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-home\") pod \"5f206f09-31fd-4bad-9637-28743bbd7a78\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") "
Sep 07 11:45:23.431835 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.431708    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcp5v\" (UniqueName: \"kubernetes.io/projected/5f206f09-31fd-4bad-9637-28743bbd7a78-kube-api-access-wcp5v\") pod \"5f206f09-31fd-4bad-9637-28743bbd7a78\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") "
Sep 07 11:45:23.431835 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.431727    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-tmp-dir\") pod \"5f206f09-31fd-4bad-9637-28743bbd7a78\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") "
Sep 07 11:45:23.431903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.431847    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f206f09-31fd-4bad-9637-28743bbd7a78-tls-certs\") pod \"5f206f09-31fd-4bad-9637-28743bbd7a78\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") "
Sep 07 11:45:23.431903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.431882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-model-cache\") pod \"5f206f09-31fd-4bad-9637-28743bbd7a78\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") "
Sep 07 11:45:23.432005 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.431911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-dshm\") pod \"5f206f09-31fd-4bad-9637-28743bbd7a78\" (UID: \"5f206f09-31fd-4bad-9637-28743bbd7a78\") "
Sep 07 11:45:23.432057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.431908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-home" (OuterVolumeSpecName: "home") pod "5f206f09-31fd-4bad-9637-28743bbd7a78" (UID: "5f206f09-31fd-4bad-9637-28743bbd7a78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:23.432057 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.432013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f206f09-31fd-4bad-9637-28743bbd7a78" (UID: "5f206f09-31fd-4bad-9637-28743bbd7a78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:23.432166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.432142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f206f09-31fd-4bad-9637-28743bbd7a78" (UID: "5f206f09-31fd-4bad-9637-28743bbd7a78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:23.432253 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.432233    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:23.432302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.432259    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:23.432302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.432277    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:23.433914 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.433894    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-dshm" (OuterVolumeSpecName: "dshm") pod "5f206f09-31fd-4bad-9637-28743bbd7a78" (UID: "5f206f09-31fd-4bad-9637-28743bbd7a78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:45:23.433978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.433957    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f206f09-31fd-4bad-9637-28743bbd7a78-kube-api-access-wcp5v" (OuterVolumeSpecName: "kube-api-access-wcp5v") pod "5f206f09-31fd-4bad-9637-28743bbd7a78" (UID: "5f206f09-31fd-4bad-9637-28743bbd7a78"). InnerVolumeSpecName "kube-api-access-wcp5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:45:23.434048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.434028    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f206f09-31fd-4bad-9637-28743bbd7a78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f206f09-31fd-4bad-9637-28743bbd7a78" (UID: "5f206f09-31fd-4bad-9637-28743bbd7a78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:45:23.532891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.532819    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f206f09-31fd-4bad-9637-28743bbd7a78-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:23.532891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.532844    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f206f09-31fd-4bad-9637-28743bbd7a78-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:23.532891 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.532857    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcp5v\" (UniqueName: \"kubernetes.io/projected/5f206f09-31fd-4bad-9637-28743bbd7a78-kube-api-access-wcp5v\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:45:23.558420 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.558390    2578 generic.go:356] "Generic (PLEG): container finished" podID="5f206f09-31fd-4bad-9637-28743bbd7a78" containerID="32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1" exitCode=0
Sep 07 11:45:23.558530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.558437    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"
Sep 07 11:45:23.558530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.558484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn" event={"ID":"5f206f09-31fd-4bad-9637-28743bbd7a78","Type":"ContainerDied","Data":"32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1"}
Sep 07 11:45:23.558530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.558521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn" event={"ID":"5f206f09-31fd-4bad-9637-28743bbd7a78","Type":"ContainerDied","Data":"569bc8e044c0f8c8b1a9d9d6a0c9f5fa09ebc321dfeece43b54705532293aa49"}
Sep 07 11:45:23.558705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.558538    2578 scope.go:117] "RemoveContainer" containerID="32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1"
Sep 07 11:45:23.567984 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.567968    2578 scope.go:117] "RemoveContainer" containerID="32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1"
Sep 07 11:45:23.568254 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:45:23.568237    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1\": container with ID starting with 32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1 not found: ID does not exist" containerID="32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1"
Sep 07 11:45:23.568301 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.568261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1"} err="failed to get container status \"32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1\": rpc error: code = NotFound desc = could not find container \"32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1\": container with ID starting with 32fc949d6db2d6086c12a05c5b893ec3134f944ae0b2977cfe64bf9c9ee54be1 not found: ID does not exist"
Sep 07 11:45:23.580326 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.580300    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"]
Sep 07 11:45:23.584191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:23.584168    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/auth-test-model-0f0a2f-kserve-6d98566b47-mr9nn"]
Sep 07 11:45:24.704443 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704406    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"]
Sep 07 11:45:24.704820 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704806    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f206f09-31fd-4bad-9637-28743bbd7a78" containerName="main"
Sep 07 11:45:24.704866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704822    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f206f09-31fd-4bad-9637-28743bbd7a78" containerName="main"
Sep 07 11:45:24.704866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704832    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a0a941-54a7-49c8-9ae3-101effc73dbf" containerName="cleanup"
Sep 07 11:45:24.704866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704837    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a0a941-54a7-49c8-9ae3-101effc73dbf" containerName="cleanup"
Sep 07 11:45:24.704866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704850    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3e2cdd-1090-43c3-bef4-d74b409e3279" containerName="main"
Sep 07 11:45:24.704866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704856    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3e2cdd-1090-43c3-bef4-d74b409e3279" containerName="main"
Sep 07 11:45:24.704866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704862    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8ebff3-be57-405e-9744-74ba1dabaf56" containerName="main"
Sep 07 11:45:24.704866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704868    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8ebff3-be57-405e-9744-74ba1dabaf56" containerName="main"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704874    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3bdcf7-5b17-490e-a904-c287a76a236a" containerName="cleanup"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704879    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3bdcf7-5b17-490e-a904-c287a76a236a" containerName="cleanup"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704886    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aafe403-36ae-4bbd-9f77-dc0c506e71cf" containerName="cleanup"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704890    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aafe403-36ae-4bbd-9f77-dc0c506e71cf" containerName="cleanup"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704896    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67c82285-dbfd-453b-aea0-fdbfe519923b" containerName="mt-curl-7224-062991"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704902    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67c82285-dbfd-453b-aea0-fdbfe519923b" containerName="mt-curl-7224-062991"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704958    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8ebff3-be57-405e-9744-74ba1dabaf56" containerName="main"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704965    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3bdcf7-5b17-490e-a904-c287a76a236a" containerName="cleanup"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704974    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3e2cdd-1090-43c3-bef4-d74b409e3279" containerName="main"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704979    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f206f09-31fd-4bad-9637-28743bbd7a78" containerName="main"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704984    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a0a941-54a7-49c8-9ae3-101effc73dbf" containerName="cleanup"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704991    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aafe403-36ae-4bbd-9f77-dc0c506e71cf" containerName="cleanup"
Sep 07 11:45:24.705083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.704998    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67c82285-dbfd-453b-aea0-fdbfe519923b" containerName="mt-curl-7224-062991"
Sep 07 11:45:24.709473 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.709450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.713012 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.712987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"sub-test-model-6b944b-kserve-self-signed-certs\""
Sep 07 11:45:24.713141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.713051    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"openshift-service-ca.crt\""
Sep 07 11:45:24.713141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.713052    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"default-dockercfg-mbjkb\""
Sep 07 11:45:24.713141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.713068    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"kube-root-ca.crt\""
Sep 07 11:45:24.718360 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.718335    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"]
Sep 07 11:45:24.798954 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.798919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:24.845325 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.845288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9xg\" (UniqueName: \"kubernetes.io/projected/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-kube-api-access-7s9xg\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.845459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.845341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-model-cache\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.845459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.845414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tmp-dir\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.845459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.845441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tls-certs\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.845714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.845547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-home\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.845714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.845589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-dshm\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.946550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.946516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-home\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.946550 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.946549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-dshm\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.946733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.946598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9xg\" (UniqueName: \"kubernetes.io/projected/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-kube-api-access-7s9xg\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.946733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.946624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-model-cache\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.946733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.946644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tmp-dir\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.946733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.946661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tls-certs\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.946935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.946899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-home\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.947033 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.947002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-model-cache\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.947130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.947049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tmp-dir\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.948804 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.948786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-dshm\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.949199 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.949179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tls-certs\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.957291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.957237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9xg\" (UniqueName: \"kubernetes.io/projected/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-kube-api-access-7s9xg\") pod \"sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:24.961603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:24.961577    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f206f09-31fd-4bad-9637-28743bbd7a78" path="/var/lib/kubelet/pods/5f206f09-31fd-4bad-9637-28743bbd7a78/volumes"
Sep 07 11:45:25.020123 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:25.020099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:25.203232 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:25.203195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:25.295899 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:25.295871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"]
Sep 07 11:45:25.297434 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:45:25.297408    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50fa1d7d_f786_4d66_a1a0_159b9136c1e4.slice/crio-c92a39bbbf658fae36ce5fe8254a8f81a1f8cef8b79a793413efbbcc1f3d9534 WatchSource:0}: Error finding container c92a39bbbf658fae36ce5fe8254a8f81a1f8cef8b79a793413efbbcc1f3d9534: Status 404 returned error can't find the container with id c92a39bbbf658fae36ce5fe8254a8f81a1f8cef8b79a793413efbbcc1f3d9534
Sep 07 11:45:25.573786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:25.573693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz" event={"ID":"50fa1d7d-f786-4d66-a1a0-159b9136c1e4","Type":"ContainerStarted","Data":"ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82"}
Sep 07 11:45:25.573786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:25.573734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz" event={"ID":"50fa1d7d-f786-4d66-a1a0-159b9136c1e4","Type":"ContainerStarted","Data":"c92a39bbbf658fae36ce5fe8254a8f81a1f8cef8b79a793413efbbcc1f3d9534"}
Sep 07 11:45:25.592261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:25.592198    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz" podStartSLOduration=1.592180955 podStartE2EDuration="1.592180955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:45:25.591792469 +0000 UTC m=+1357.066640645" watchObservedRunningTime="2026-09-07 11:45:25.592180955 +0000 UTC m=+1357.067029104"
Sep 07 11:45:27.073679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:27.073649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:29.308201 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:29.308165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:45:33.064287 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:33.064251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:33.128214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:33.128182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:33.350708 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:33.350628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:34.296965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:34.296924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:34.695137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:34.695098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:34.794013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:34.793979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:34.893656 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:34.893625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:35.020723 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:35.020643    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:35.020926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:35.020907    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:35.033579 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:35.033547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:35.557983 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:35.557949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:35.646981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:35.646953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:45:35.654289 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:35.654253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:35.753902 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:35.753875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:37.089106 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:37.089072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:43.133579 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:43.133543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:43.310558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:43.310526    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:43.834664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:43.834635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:49.708291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:49.708255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:49.812116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:49.812087    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:49.909862 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:49.909822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:52.300426 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.300396    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"]
Sep 07 11:45:52.305405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.305380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.308931 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.308908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"default-dockercfg-8qzws\""
Sep 07 11:45:52.309075 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.308959    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"kube-root-ca.crt\""
Sep 07 11:45:52.309075 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.308987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"sub-test-model-0f0a2f-kserve-self-signed-certs\""
Sep 07 11:45:52.309075 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.309030    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"openshift-service-ca.crt\""
Sep 07 11:45:52.320895 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.320868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"]
Sep 07 11:45:52.342572 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.342531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:45:52.491287 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.491254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-tmp-dir\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.491438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.491291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmmg\" (UniqueName: \"kubernetes.io/projected/ee12d343-36bb-4919-bec3-e036a1f62452-kube-api-access-qvmmg\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.491438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.491319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee12d343-36bb-4919-bec3-e036a1f62452-tls-certs\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.491555 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.491452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-dshm\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.491555 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.491508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-model-cache\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.491555 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.491538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-home\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.592658 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.592593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-dshm\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.592658 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.592626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-model-cache\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.592658 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.592645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-home\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.592866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.592712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-tmp-dir\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.592866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.592741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmmg\" (UniqueName: \"kubernetes.io/projected/ee12d343-36bb-4919-bec3-e036a1f62452-kube-api-access-qvmmg\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.592866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.592774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee12d343-36bb-4919-bec3-e036a1f62452-tls-certs\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.593115 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.593090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-model-cache\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.593187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.593126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-home\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.593187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.593175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-tmp-dir\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.594877 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.594858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-dshm\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.595161 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.595143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee12d343-36bb-4919-bec3-e036a1f62452-tls-certs\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.601583 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.601560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmmg\" (UniqueName: \"kubernetes.io/projected/ee12d343-36bb-4919-bec3-e036a1f62452-kube-api-access-qvmmg\") pod \"sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.625237 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.625213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:45:52.897917 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:52.897885    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"]
Sep 07 11:45:52.898833 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:45:52.898804    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee12d343_36bb_4919_bec3_e036a1f62452.slice/crio-248c2a838eb189dbb614a9c99f2d473090e90b413dcdab6530f5b1bfaf3d21e2 WatchSource:0}: Error finding container 248c2a838eb189dbb614a9c99f2d473090e90b413dcdab6530f5b1bfaf3d21e2: Status 404 returned error can't find the container with id 248c2a838eb189dbb614a9c99f2d473090e90b413dcdab6530f5b1bfaf3d21e2
Sep 07 11:45:53.749012 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:53.748975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8" event={"ID":"ee12d343-36bb-4919-bec3-e036a1f62452","Type":"ContainerStarted","Data":"27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad"}
Sep 07 11:45:53.749012 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:53.749015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8" event={"ID":"ee12d343-36bb-4919-bec3-e036a1f62452","Type":"ContainerStarted","Data":"248c2a838eb189dbb614a9c99f2d473090e90b413dcdab6530f5b1bfaf3d21e2"}
Sep 07 11:45:53.767455 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:53.767373    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8" podStartSLOduration=1.767359013 podStartE2EDuration="1.767359013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:45:53.766634378 +0000 UTC m=+1385.241482550" watchObservedRunningTime="2026-09-07 11:45:53.767359013 +0000 UTC m=+1385.242207161"
Sep 07 11:45:55.900658 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:45:55.900625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:46:00.471337 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:00.471302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:00.703128 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:00.703092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:02.626290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:02.626255    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:46:02.626710 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:02.626304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:46:02.639144 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:02.639113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:46:02.811805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:02.811778    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:46:05.459591 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:05.459556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:05.560290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:05.560257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:10.593742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:10.593707    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:46:21.650021 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:21.649986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:23.892751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:23.892715    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7224-399f16"]
Sep 07 11:46:23.896372 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:23.896353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:23.899001 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:23.898981    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhmxq\""
Sep 07 11:46:23.905371 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:23.905346    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7224-399f16"]
Sep 07 11:46:23.962330 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:23.962297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qz9h\" (UniqueName: \"kubernetes.io/projected/3c5a790b-b6fc-406f-b735-eadcf60e69a9-kube-api-access-8qz9h\") pod \"mt-curl-7224-399f16\" (UID: \"3c5a790b-b6fc-406f-b735-eadcf60e69a9\") " pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:24.063643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:24.063611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qz9h\" (UniqueName: \"kubernetes.io/projected/3c5a790b-b6fc-406f-b735-eadcf60e69a9-kube-api-access-8qz9h\") pod \"mt-curl-7224-399f16\" (UID: \"3c5a790b-b6fc-406f-b735-eadcf60e69a9\") " pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:24.071804 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:24.071782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qz9h\" (UniqueName: \"kubernetes.io/projected/3c5a790b-b6fc-406f-b735-eadcf60e69a9-kube-api-access-8qz9h\") pod \"mt-curl-7224-399f16\" (UID: \"3c5a790b-b6fc-406f-b735-eadcf60e69a9\") " pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:24.207241 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:24.207169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:24.479824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:24.479800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7224-399f16"]
Sep 07 11:46:24.484278 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:46:24.484195    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c5a790b_b6fc_406f_b735_eadcf60e69a9.slice/crio-b0e4c32aa07a11eb03bf7fd7c12ef784acb0d36b0b4941aba912dc3e37d28b0f WatchSource:0}: Error finding container b0e4c32aa07a11eb03bf7fd7c12ef784acb0d36b0b4941aba912dc3e37d28b0f: Status 404 returned error can't find the container with id b0e4c32aa07a11eb03bf7fd7c12ef784acb0d36b0b4941aba912dc3e37d28b0f
Sep 07 11:46:24.933169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:24.933138    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c5a790b-b6fc-406f-b735-eadcf60e69a9" containerID="f318ff3e26120051a1e88db26fc2228c9a96f6993c84f9b093c14ce35b0d59c0" exitCode=0
Sep 07 11:46:24.933502 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:24.933206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7224-399f16" event={"ID":"3c5a790b-b6fc-406f-b735-eadcf60e69a9","Type":"ContainerDied","Data":"f318ff3e26120051a1e88db26fc2228c9a96f6993c84f9b093c14ce35b0d59c0"}
Sep 07 11:46:24.933502 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:24.933245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7224-399f16" event={"ID":"3c5a790b-b6fc-406f-b735-eadcf60e69a9","Type":"ContainerStarted","Data":"b0e4c32aa07a11eb03bf7fd7c12ef784acb0d36b0b4941aba912dc3e37d28b0f"}
Sep 07 11:46:26.072742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.072720    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:26.097186 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.097166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7224-399f16_3c5a790b-b6fc-406f-b735-eadcf60e69a9/mt-curl-7224-399f16/0.log"
Sep 07 11:46:26.125817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.125793    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7224-399f16"]
Sep 07 11:46:26.129392 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.129369    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7224-399f16"]
Sep 07 11:46:26.182798 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.182773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qz9h\" (UniqueName: \"kubernetes.io/projected/3c5a790b-b6fc-406f-b735-eadcf60e69a9-kube-api-access-8qz9h\") pod \"3c5a790b-b6fc-406f-b735-eadcf60e69a9\" (UID: \"3c5a790b-b6fc-406f-b735-eadcf60e69a9\") "
Sep 07 11:46:26.184692 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.184658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5a790b-b6fc-406f-b735-eadcf60e69a9-kube-api-access-8qz9h" (OuterVolumeSpecName: "kube-api-access-8qz9h") pod "3c5a790b-b6fc-406f-b735-eadcf60e69a9" (UID: "3c5a790b-b6fc-406f-b735-eadcf60e69a9"). InnerVolumeSpecName "kube-api-access-8qz9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:46:26.284223 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.284199    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qz9h\" (UniqueName: \"kubernetes.io/projected/3c5a790b-b6fc-406f-b735-eadcf60e69a9-kube-api-access-8qz9h\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:26.306864 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.306836    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7224-767e43"]
Sep 07 11:46:26.307243 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.307230    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5a790b-b6fc-406f-b735-eadcf60e69a9" containerName="mt-curl-7224-399f16"
Sep 07 11:46:26.307286 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.307245    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5a790b-b6fc-406f-b735-eadcf60e69a9" containerName="mt-curl-7224-399f16"
Sep 07 11:46:26.307335 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.307326    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5a790b-b6fc-406f-b735-eadcf60e69a9" containerName="mt-curl-7224-399f16"
Sep 07 11:46:26.310407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.310391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:26.327020 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.326998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7224-767e43"]
Sep 07 11:46:26.385578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.385549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxx8\" (UniqueName: \"kubernetes.io/projected/53c0a56b-4865-4ed3-a100-5236bfacaede-kube-api-access-hfxx8\") pod \"mt-curl-7224-767e43\" (UID: \"53c0a56b-4865-4ed3-a100-5236bfacaede\") " pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:26.486204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.486131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxx8\" (UniqueName: \"kubernetes.io/projected/53c0a56b-4865-4ed3-a100-5236bfacaede-kube-api-access-hfxx8\") pod \"mt-curl-7224-767e43\" (UID: \"53c0a56b-4865-4ed3-a100-5236bfacaede\") " pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:26.494978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.494954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxx8\" (UniqueName: \"kubernetes.io/projected/53c0a56b-4865-4ed3-a100-5236bfacaede-kube-api-access-hfxx8\") pod \"mt-curl-7224-767e43\" (UID: \"53c0a56b-4865-4ed3-a100-5236bfacaede\") " pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:26.619265 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.619242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:26.892105 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.892078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7224-767e43"]
Sep 07 11:46:26.894067 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:46:26.894043    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53c0a56b_4865_4ed3_a100_5236bfacaede.slice/crio-fd22e957be23ea7ff196830a84816c2ce14008febd178b48e76fa864541fdfa1 WatchSource:0}: Error finding container fd22e957be23ea7ff196830a84816c2ce14008febd178b48e76fa864541fdfa1: Status 404 returned error can't find the container with id fd22e957be23ea7ff196830a84816c2ce14008febd178b48e76fa864541fdfa1
Sep 07 11:46:26.948783 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.948760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:26.948890 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.948767    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0e4c32aa07a11eb03bf7fd7c12ef784acb0d36b0b4941aba912dc3e37d28b0f"
Sep 07 11:46:26.950186 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.950158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7224-767e43" event={"ID":"53c0a56b-4865-4ed3-a100-5236bfacaede","Type":"ContainerStarted","Data":"fd22e957be23ea7ff196830a84816c2ce14008febd178b48e76fa864541fdfa1"}
Sep 07 11:46:26.960613 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:46:26.960588    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7224-399f16\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="3c5a790b-b6fc-406f-b735-eadcf60e69a9" pod="opendatahub/mt-curl-7224-399f16"
Sep 07 11:46:26.961546 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:26.961524    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5a790b-b6fc-406f-b735-eadcf60e69a9" path="/var/lib/kubelet/pods/3c5a790b-b6fc-406f-b735-eadcf60e69a9/volumes"
Sep 07 11:46:27.900836 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:27.900798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:46:27.957242 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:27.957211    2578 generic.go:356] "Generic (PLEG): container finished" podID="53c0a56b-4865-4ed3-a100-5236bfacaede" containerID="4f147a0bcdd57483ca4a978809f3d3d2f9f1d5dfafaf44c21aef3bba035b034c" exitCode=0
Sep 07 11:46:27.957402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:27.957301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7224-767e43" event={"ID":"53c0a56b-4865-4ed3-a100-5236bfacaede","Type":"ContainerDied","Data":"4f147a0bcdd57483ca4a978809f3d3d2f9f1d5dfafaf44c21aef3bba035b034c"}
Sep 07 11:46:27.960259 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:27.960237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:28.059915 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:28.059892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:28.324707 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:28.324626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:28.680941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:28.680907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:29.102094 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.102069    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:29.126004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.125981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7224-767e43_53c0a56b-4865-4ed3-a100-5236bfacaede/mt-curl-7224-767e43/0.log"
Sep 07 11:46:29.149632 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.149607    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7224-767e43"]
Sep 07 11:46:29.153003 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.152980    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7224-767e43"]
Sep 07 11:46:29.209176 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.209148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfxx8\" (UniqueName: \"kubernetes.io/projected/53c0a56b-4865-4ed3-a100-5236bfacaede-kube-api-access-hfxx8\") pod \"53c0a56b-4865-4ed3-a100-5236bfacaede\" (UID: \"53c0a56b-4865-4ed3-a100-5236bfacaede\") "
Sep 07 11:46:29.211216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.211194    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c0a56b-4865-4ed3-a100-5236bfacaede-kube-api-access-hfxx8" (OuterVolumeSpecName: "kube-api-access-hfxx8") pod "53c0a56b-4865-4ed3-a100-5236bfacaede" (UID: "53c0a56b-4865-4ed3-a100-5236bfacaede"). InnerVolumeSpecName "kube-api-access-hfxx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:46:29.272238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.272210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:29.310769 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.310743    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfxx8\" (UniqueName: \"kubernetes.io/projected/53c0a56b-4865-4ed3-a100-5236bfacaede-kube-api-access-hfxx8\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:29.969964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.969929    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd22e957be23ea7ff196830a84816c2ce14008febd178b48e76fa864541fdfa1"
Sep 07 11:46:29.969964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:29.969961    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:29.980108 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:46:29.980078    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7224-767e43\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="53c0a56b-4865-4ed3-a100-5236bfacaede" pod="opendatahub/mt-curl-7224-767e43"
Sep 07 11:46:30.960422 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:30.960378    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c0a56b-4865-4ed3-a100-5236bfacaede" path="/var/lib/kubelet/pods/53c0a56b-4865-4ed3-a100-5236bfacaede/volumes"
Sep 07 11:46:31.632324 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:31.632283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:31.733083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:31.733052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:31.980661 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:31.980577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"]
Sep 07 11:46:31.998822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:31.998791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:32.814417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:32.814386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:32.912564 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:32.912533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:33.036239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.036208    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:33.077587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.077514    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"]
Sep 07 11:46:33.099361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.099309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:33.401372 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.401328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:33.502555 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.502524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:33.512422 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.512395    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"]
Sep 07 11:46:33.512799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.512786    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c0a56b-4865-4ed3-a100-5236bfacaede" containerName="mt-curl-7224-767e43"
Sep 07 11:46:33.512843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.512800    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c0a56b-4865-4ed3-a100-5236bfacaede" containerName="mt-curl-7224-767e43"
Sep 07 11:46:33.512899 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.512890    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c0a56b-4865-4ed3-a100-5236bfacaede" containerName="mt-curl-7224-767e43"
Sep 07 11:46:33.516291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.516275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:33.518795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.518771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2cfab4-dockercfg-zxhxq\""
Sep 07 11:46:33.518908 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.518825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2cfab4\""
Sep 07 11:46:33.525902 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.525883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"]
Sep 07 11:46:33.648527 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.648485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:33.648527 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.648520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvphw\" (UniqueName: \"kubernetes.io/projected/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-kube-api-access-wvphw\") pod \"payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:33.749824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.749746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:33.749824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.749785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvphw\" (UniqueName: \"kubernetes.io/projected/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-kube-api-access-wvphw\") pod \"payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:33.750440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.750419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:33.757639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.757610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvphw\" (UniqueName: \"kubernetes.io/projected/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-kube-api-access-wvphw\") pod \"payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:33.827954 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:33.827929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:34.101284 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:34.101258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"]
Sep 07 11:46:34.102866 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:46:34.102840    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d6e7ee_3311_4385_9b1a_b5b0f407c52e.slice/crio-7b300b47038b9a70515eaf75fa60a1cfa2a481bf3ba2735cd29c5678336ea841 WatchSource:0}: Error finding container 7b300b47038b9a70515eaf75fa60a1cfa2a481bf3ba2735cd29c5678336ea841: Status 404 returned error can't find the container with id 7b300b47038b9a70515eaf75fa60a1cfa2a481bf3ba2735cd29c5678336ea841
Sep 07 11:46:35.001015 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:35.000974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97" event={"ID":"98d6e7ee-3311-4385-9b1a-b5b0f407c52e","Type":"ContainerStarted","Data":"e10a5bf822d139fe2f2f66a626c870876af0c80ac65ed7d9b8749668f528ed78"}
Sep 07 11:46:35.001187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:35.001026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97" event={"ID":"98d6e7ee-3311-4385-9b1a-b5b0f407c52e","Type":"ContainerStarted","Data":"7b300b47038b9a70515eaf75fa60a1cfa2a481bf3ba2735cd29c5678336ea841"}
Sep 07 11:46:35.001187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:35.001054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:35.020601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:35.020550    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97" podStartSLOduration=2.020534667 podStartE2EDuration="2.020534667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:46:35.018089074 +0000 UTC m=+1426.492937245" watchObservedRunningTime="2026-09-07 11:46:35.020534667 +0000 UTC m=+1426.495382816"
Sep 07 11:46:40.692627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.692587    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"]
Sep 07 11:46:40.696574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.696551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.698871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.698850    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b944b\"/\"rate-test-model-6b944b-kserve-self-signed-certs\""
Sep 07 11:46:40.704669 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.704646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"]
Sep 07 11:46:40.734369 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.734315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:40.814004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.813970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-dshm\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.814138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.814037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-tmp-dir\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.814138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.814072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415452db-c037-4b66-83b7-8315cfb84bbe-tls-certs\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.814138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.814088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-home\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.814291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.814160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-model-cache\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.814291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.814205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5m79\" (UniqueName: \"kubernetes.io/projected/415452db-c037-4b66-83b7-8315cfb84bbe-kube-api-access-r5m79\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.915660 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.915626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-tmp-dir\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.915824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.915666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415452db-c037-4b66-83b7-8315cfb84bbe-tls-certs\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.915824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.915694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-home\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.915824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.915749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-model-cache\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.915824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.915798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5m79\" (UniqueName: \"kubernetes.io/projected/415452db-c037-4b66-83b7-8315cfb84bbe-kube-api-access-r5m79\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.916044 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.915862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-dshm\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.916149 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.916123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-tmp-dir\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.916285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.916162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-home\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.916285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.916174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-model-cache\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.918050 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.918030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-dshm\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.918251 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.918230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415452db-c037-4b66-83b7-8315cfb84bbe-tls-certs\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:40.924476 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:40.924440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5m79\" (UniqueName: \"kubernetes.io/projected/415452db-c037-4b66-83b7-8315cfb84bbe-kube-api-access-r5m79\") pod \"rate-test-model-6b944b-kserve-6458cf9df9-r6mqg\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") " pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:41.007284 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:41.007227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:41.282069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:41.282044    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"]
Sep 07 11:46:41.284025 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:46:41.283984    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod415452db_c037_4b66_83b7_8315cfb84bbe.slice/crio-90078d78e6b12a0aa9b1e8cec4cc1c163c0a8fe7f6ec772da4fe7648b52a3508 WatchSource:0}: Error finding container 90078d78e6b12a0aa9b1e8cec4cc1c163c0a8fe7f6ec772da4fe7648b52a3508: Status 404 returned error can't find the container with id 90078d78e6b12a0aa9b1e8cec4cc1c163c0a8fe7f6ec772da4fe7648b52a3508
Sep 07 11:46:42.049430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:42.049352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg" event={"ID":"415452db-c037-4b66-83b7-8315cfb84bbe","Type":"ContainerStarted","Data":"149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4"}
Sep 07 11:46:42.049430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:42.049402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg" event={"ID":"415452db-c037-4b66-83b7-8315cfb84bbe","Type":"ContainerStarted","Data":"90078d78e6b12a0aa9b1e8cec4cc1c163c0a8fe7f6ec772da4fe7648b52a3508"}
Sep 07 11:46:42.070638 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:42.070590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg" podStartSLOduration=2.070576921 podStartE2EDuration="2.070576921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:46:42.068126199 +0000 UTC m=+1433.542974374" watchObservedRunningTime="2026-09-07 11:46:42.070576921 +0000 UTC m=+1433.545425069"
Sep 07 11:46:44.113073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:44.113036    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:46:44.567046 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:44.566965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:46.008641 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:46.008611    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:46:47.000085 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.000047    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz" podUID="50fa1d7d-f786-4d66-a1a0-159b9136c1e4" containerName="main" containerID="cri-o://ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82" gracePeriod=45
Sep 07 11:46:47.242906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.242883    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:46:47.275585 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275518    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-home\") pod \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") "
Sep 07 11:46:47.275585 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-model-cache\") pod \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") "
Sep 07 11:46:47.275742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-dshm\") pod \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") "
Sep 07 11:46:47.275742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tmp-dir\") pod \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") "
Sep 07 11:46:47.275742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s9xg\" (UniqueName: \"kubernetes.io/projected/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-kube-api-access-7s9xg\") pod \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") "
Sep 07 11:46:47.275742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tls-certs\") pod \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\" (UID: \"50fa1d7d-f786-4d66-a1a0-159b9136c1e4\") "
Sep 07 11:46:47.275959 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-home" (OuterVolumeSpecName: "home") pod "50fa1d7d-f786-4d66-a1a0-159b9136c1e4" (UID: "50fa1d7d-f786-4d66-a1a0-159b9136c1e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:47.275959 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "50fa1d7d-f786-4d66-a1a0-159b9136c1e4" (UID: "50fa1d7d-f786-4d66-a1a0-159b9136c1e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:47.275959 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.275920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50fa1d7d-f786-4d66-a1a0-159b9136c1e4" (UID: "50fa1d7d-f786-4d66-a1a0-159b9136c1e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:47.276544 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.276519    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:47.276651 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.276546    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:47.276651 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.276581    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:47.281501 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.278850    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50fa1d7d-f786-4d66-a1a0-159b9136c1e4" (UID: "50fa1d7d-f786-4d66-a1a0-159b9136c1e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:46:47.281501 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.279106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-kube-api-access-7s9xg" (OuterVolumeSpecName: "kube-api-access-7s9xg") pod "50fa1d7d-f786-4d66-a1a0-159b9136c1e4" (UID: "50fa1d7d-f786-4d66-a1a0-159b9136c1e4"). InnerVolumeSpecName "kube-api-access-7s9xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:46:47.286321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.286271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-dshm" (OuterVolumeSpecName: "dshm") pod "50fa1d7d-f786-4d66-a1a0-159b9136c1e4" (UID: "50fa1d7d-f786-4d66-a1a0-159b9136c1e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:47.377568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.377542    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9xg\" (UniqueName: \"kubernetes.io/projected/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-kube-api-access-7s9xg\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:47.377568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.377565    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:47.377714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:47.377575    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa1d7d-f786-4d66-a1a0-159b9136c1e4-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:48.086709 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.086672    2578 generic.go:356] "Generic (PLEG): container finished" podID="50fa1d7d-f786-4d66-a1a0-159b9136c1e4" containerID="ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82" exitCode=0
Sep 07 11:46:48.086877 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.086728    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"
Sep 07 11:46:48.086877 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.086761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz" event={"ID":"50fa1d7d-f786-4d66-a1a0-159b9136c1e4","Type":"ContainerDied","Data":"ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82"}
Sep 07 11:46:48.086877 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.086802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz" event={"ID":"50fa1d7d-f786-4d66-a1a0-159b9136c1e4","Type":"ContainerDied","Data":"c92a39bbbf658fae36ce5fe8254a8f81a1f8cef8b79a793413efbbcc1f3d9534"}
Sep 07 11:46:48.086877 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.086819    2578 scope.go:117] "RemoveContainer" containerID="ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82"
Sep 07 11:46:48.097253 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.097236    2578 scope.go:117] "RemoveContainer" containerID="ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82"
Sep 07 11:46:48.097536 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:46:48.097512    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82\": container with ID starting with ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82 not found: ID does not exist" containerID="ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82"
Sep 07 11:46:48.097630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.097542    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82"} err="failed to get container status \"ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82\": rpc error: code = NotFound desc = could not find container \"ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82\": container with ID starting with ce8db6d860c7076965f0a51de63619c4e3d23539cbd7ddd1d3fbda350d5d9b82 not found: ID does not exist"
Sep 07 11:46:48.098846 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.098824    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8" podUID="ee12d343-36bb-4919-bec3-e036a1f62452" containerName="main" containerID="cri-o://27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad" gracePeriod=45
Sep 07 11:46:48.109604 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.109579    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"]
Sep 07 11:46:48.114711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.114693    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/sub-test-model-6b944b-kserve-66f8dcbd8f-p6kdz"]
Sep 07 11:46:48.344419 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.344358    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:46:48.386608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-home\") pod \"ee12d343-36bb-4919-bec3-e036a1f62452\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") "
Sep 07 11:46:48.386753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-tmp-dir\") pod \"ee12d343-36bb-4919-bec3-e036a1f62452\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") "
Sep 07 11:46:48.386753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvmmg\" (UniqueName: \"kubernetes.io/projected/ee12d343-36bb-4919-bec3-e036a1f62452-kube-api-access-qvmmg\") pod \"ee12d343-36bb-4919-bec3-e036a1f62452\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") "
Sep 07 11:46:48.386753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee12d343-36bb-4919-bec3-e036a1f62452-tls-certs\") pod \"ee12d343-36bb-4919-bec3-e036a1f62452\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") "
Sep 07 11:46:48.386917 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-dshm\") pod \"ee12d343-36bb-4919-bec3-e036a1f62452\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") "
Sep 07 11:46:48.386917 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-model-cache\") pod \"ee12d343-36bb-4919-bec3-e036a1f62452\" (UID: \"ee12d343-36bb-4919-bec3-e036a1f62452\") "
Sep 07 11:46:48.386917 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386830    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-home" (OuterVolumeSpecName: "home") pod "ee12d343-36bb-4919-bec3-e036a1f62452" (UID: "ee12d343-36bb-4919-bec3-e036a1f62452"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:48.387083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.386945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee12d343-36bb-4919-bec3-e036a1f62452" (UID: "ee12d343-36bb-4919-bec3-e036a1f62452"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:48.387140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.387100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee12d343-36bb-4919-bec3-e036a1f62452" (UID: "ee12d343-36bb-4919-bec3-e036a1f62452"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:48.387284 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.387265    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:48.387357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.387290    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:48.387357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.387306    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:48.388808 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.388784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee12d343-36bb-4919-bec3-e036a1f62452-kube-api-access-qvmmg" (OuterVolumeSpecName: "kube-api-access-qvmmg") pod "ee12d343-36bb-4919-bec3-e036a1f62452" (UID: "ee12d343-36bb-4919-bec3-e036a1f62452"). InnerVolumeSpecName "kube-api-access-qvmmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:46:48.388903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.388815    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-dshm" (OuterVolumeSpecName: "dshm") pod "ee12d343-36bb-4919-bec3-e036a1f62452" (UID: "ee12d343-36bb-4919-bec3-e036a1f62452"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:46:48.388903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.388857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee12d343-36bb-4919-bec3-e036a1f62452-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee12d343-36bb-4919-bec3-e036a1f62452" (UID: "ee12d343-36bb-4919-bec3-e036a1f62452"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:46:48.488171 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.488148    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmmg\" (UniqueName: \"kubernetes.io/projected/ee12d343-36bb-4919-bec3-e036a1f62452-kube-api-access-qvmmg\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:48.488171 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.488170    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee12d343-36bb-4919-bec3-e036a1f62452-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:48.488321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.488179    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee12d343-36bb-4919-bec3-e036a1f62452-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:46:48.960933 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:48.960898    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50fa1d7d-f786-4d66-a1a0-159b9136c1e4" path="/var/lib/kubelet/pods/50fa1d7d-f786-4d66-a1a0-159b9136c1e4/volumes"
Sep 07 11:46:49.094362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.094331    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee12d343-36bb-4919-bec3-e036a1f62452" containerID="27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad" exitCode=0
Sep 07 11:46:49.094536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.094370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8" event={"ID":"ee12d343-36bb-4919-bec3-e036a1f62452","Type":"ContainerDied","Data":"27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad"}
Sep 07 11:46:49.094536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.094395    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"
Sep 07 11:46:49.094536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.094420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8" event={"ID":"ee12d343-36bb-4919-bec3-e036a1f62452","Type":"ContainerDied","Data":"248c2a838eb189dbb614a9c99f2d473090e90b413dcdab6530f5b1bfaf3d21e2"}
Sep 07 11:46:49.094536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.094442    2578 scope.go:117] "RemoveContainer" containerID="27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad"
Sep 07 11:46:49.103648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.103631    2578 scope.go:117] "RemoveContainer" containerID="27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad"
Sep 07 11:46:49.103899 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:46:49.103882    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad\": container with ID starting with 27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad not found: ID does not exist" containerID="27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad"
Sep 07 11:46:49.103944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.103906    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad"} err="failed to get container status \"27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad\": rpc error: code = NotFound desc = could not find container \"27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad\": container with ID starting with 27bc73a1ad29e2940712df5c408c8d6e2495c21adcad4c2706d534d0de9fa4ad not found: ID does not exist"
Sep 07 11:46:49.110665 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.110641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"]
Sep 07 11:46:49.116376 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.116357    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/sub-test-model-0f0a2f-kserve-c888cc7fc-89lk8"]
Sep 07 11:46:49.262184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:49.262126    2578 scope.go:117] "RemoveContainer" containerID="010199b2009cb22bed8bba31c917dc28806445a2f10d1c2a8867951d3091b2c4"
Sep 07 11:46:50.960872 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:50.960840    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee12d343-36bb-4919-bec3-e036a1f62452" path="/var/lib/kubelet/pods/ee12d343-36bb-4919-bec3-e036a1f62452/volumes"
Sep 07 11:46:51.007906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:51.007875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:51.008146 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:51.008127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:51.020778 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:51.020756    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:51.121866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:51.121838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:46:53.793269 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:53.793238    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:56.736798 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:56.736762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:57.308017 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.307977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:57.627618 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.627584    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"]
Sep 07 11:46:57.628059 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.628041    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fa1d7d-f786-4d66-a1a0-159b9136c1e4" containerName="main"
Sep 07 11:46:57.628145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.628063    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fa1d7d-f786-4d66-a1a0-159b9136c1e4" containerName="main"
Sep 07 11:46:57.628145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.628098    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee12d343-36bb-4919-bec3-e036a1f62452" containerName="main"
Sep 07 11:46:57.628145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.628107    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee12d343-36bb-4919-bec3-e036a1f62452" containerName="main"
Sep 07 11:46:57.628295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.628209    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fa1d7d-f786-4d66-a1a0-159b9136c1e4" containerName="main"
Sep 07 11:46:57.628295 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.628224    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee12d343-36bb-4919-bec3-e036a1f62452" containerName="main"
Sep 07 11:46:57.632537 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.632513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.639143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.639121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f57a16-openshift-default-dockercfg-npv5p\""
Sep 07 11:46:57.644975 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.644951    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"]
Sep 07 11:46:57.660725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.660692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:57.665791 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.665772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-socket\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.665867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.665804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e56eaae-4ca1-4cba-90fc-8080ce078127-istiod-ca-cert\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.665867 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.665821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-envoy\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.665935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.665891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-credential-socket\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.665968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.665946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-certs\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.666004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.665985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-token\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.666039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.666023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-podinfo\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.666073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.666054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-data\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.666131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.666117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6gqn\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-kube-api-access-s6gqn\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.695058 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.695034    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"]
Sep 07 11:46:57.698416 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.698401    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.700725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.700705    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"default-dockercfg-8qzws\""
Sep 07 11:46:57.700725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.700714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"openshift-service-ca.crt\""
Sep 07 11:46:57.700884 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.700779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"rate-test-model-0f0a2f-kserve-self-signed-certs\""
Sep 07 11:46:57.700884 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.700714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f0a2f\"/\"kube-root-ca.crt\""
Sep 07 11:46:57.709214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.709190    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"]
Sep 07 11:46:57.766939 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.766858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-home\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.766955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-socket\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.766992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwt5x\" (UniqueName: \"kubernetes.io/projected/a23b25d9-2669-41d6-b616-12374e2b78b7-kube-api-access-rwt5x\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.767027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e56eaae-4ca1-4cba-90fc-8080ce078127-istiod-ca-cert\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.767053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-envoy\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.767078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-model-cache\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.767101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b25d9-2669-41d6-b616-12374e2b78b7-tls-certs\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.767142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-credential-socket\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.767519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.767169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-dshm\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.768609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.768585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-credential-socket\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.769033 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.769011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-socket\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.769790 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.769745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e56eaae-4ca1-4cba-90fc-8080ce078127-istiod-ca-cert\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.769907 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.767208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-certs\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.770181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.770097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-certs\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.770296 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.770179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-token\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.770438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.770421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-podinfo\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.770581 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.770566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-data\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.770703 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.770690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-tmp-dir\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.770845 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.770831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6gqn\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-kube-api-access-s6gqn\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.770977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.770954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-data\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.772213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.772187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-envoy\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.773690 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.773649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-podinfo\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.778987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.778959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-token\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.779557 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.779533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6gqn\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-kube-api-access-s6gqn\") pod \"e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") " pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:57.801147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.801119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:57.872546 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.872516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-home\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.872701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.872560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwt5x\" (UniqueName: \"kubernetes.io/projected/a23b25d9-2669-41d6-b616-12374e2b78b7-kube-api-access-rwt5x\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.872701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.872582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-model-cache\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.872701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.872600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b25d9-2669-41d6-b616-12374e2b78b7-tls-certs\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.872701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.872621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-dshm\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.872701 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.872658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-tmp-dir\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.873002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.872981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-home\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.873058 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.873015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-model-cache\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.873097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.873068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-tmp-dir\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.874839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.874819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-dshm\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.875216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.875196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b25d9-2669-41d6-b616-12374e2b78b7-tls-certs\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.880316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.880263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwt5x\" (UniqueName: \"kubernetes.io/projected/a23b25d9-2669-41d6-b616-12374e2b78b7-kube-api-access-rwt5x\") pod \"rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") " pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:57.949895 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.949869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:46:57.951542 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:57.951527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:58.009834 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:58.009806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:46:58.228066 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:58.228041    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"]
Sep 07 11:46:58.229370 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:46:58.229346    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e56eaae_4ca1_4cba_90fc_8080ce078127.slice/crio-1373f0a7ed7cb0c71b59ffe338ca6338fb620c1179be77579d1d87b4424f05e7 WatchSource:0}: Error finding container 1373f0a7ed7cb0c71b59ffe338ca6338fb620c1179be77579d1d87b4424f05e7: Status 404 returned error can't find the container with id 1373f0a7ed7cb0c71b59ffe338ca6338fb620c1179be77579d1d87b4424f05e7
Sep 07 11:46:58.231692 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:58.231660    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:46:58.231767 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:58.231732    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:46:58.231767 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:58.231759    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:46:58.285803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:58.285780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"]
Sep 07 11:46:58.287981 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:46:58.287952    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23b25d9_2669_41d6_b616_12374e2b78b7.slice/crio-74dd8b5536b6c2bf6d1d25b16f2a213b53d3cb96c00a52a3791df63824c7ee65 WatchSource:0}: Error finding container 74dd8b5536b6c2bf6d1d25b16f2a213b53d3cb96c00a52a3791df63824c7ee65: Status 404 returned error can't find the container with id 74dd8b5536b6c2bf6d1d25b16f2a213b53d3cb96c00a52a3791df63824c7ee65
Sep 07 11:46:59.157769 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.157725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b" event={"ID":"a23b25d9-2669-41d6-b616-12374e2b78b7","Type":"ContainerStarted","Data":"95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba"}
Sep 07 11:46:59.158165 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.157784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b" event={"ID":"a23b25d9-2669-41d6-b616-12374e2b78b7","Type":"ContainerStarted","Data":"74dd8b5536b6c2bf6d1d25b16f2a213b53d3cb96c00a52a3791df63824c7ee65"}
Sep 07 11:46:59.159298 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.159271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" event={"ID":"6e56eaae-4ca1-4cba-90fc-8080ce078127","Type":"ContainerStarted","Data":"6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd"}
Sep 07 11:46:59.159430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.159318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" event={"ID":"6e56eaae-4ca1-4cba-90fc-8080ce078127","Type":"ContainerStarted","Data":"1373f0a7ed7cb0c71b59ffe338ca6338fb620c1179be77579d1d87b4424f05e7"}
Sep 07 11:46:59.178548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.178503    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b" podStartSLOduration=2.178488693 podStartE2EDuration="2.178488693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:46:59.176066893 +0000 UTC m=+1450.650915044" watchObservedRunningTime="2026-09-07 11:46:59.178488693 +0000 UTC m=+1450.653336842"
Sep 07 11:46:59.197339 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.197295    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" podStartSLOduration=2.197280656 podStartE2EDuration="2.197280656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:46:59.195029198 +0000 UTC m=+1450.669877371" watchObservedRunningTime="2026-09-07 11:46:59.197280656 +0000 UTC m=+1450.672128804"
Sep 07 11:46:59.952207 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.952176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:46:59.957213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:46:59.957187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:47:00.165143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:00.165114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:47:00.166140 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:00.166117    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:47:01.100327 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:01.100294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:47:01.257200 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:01.257166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:01.356143 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:01.356062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:02.504590 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:02.504559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:02.614197 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:02.614162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:02.712152 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:02.712114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:03.021082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.021052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:03.121361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.121329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:03.149666 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.149630    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"]
Sep 07 11:47:03.153914 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.153898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.156535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.156515    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f57a16\""
Sep 07 11:47:03.156629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.156567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f57a16-dockercfg-q5g8p\""
Sep 07 11:47:03.161611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.161589    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"]
Sep 07 11:47:03.222780 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.222747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7g2t\" (UniqueName: \"kubernetes.io/projected/11e03ce1-52cb-4d88-bb02-c513dec75d0e-kube-api-access-r7g2t\") pod \"payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.222944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.222840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e03ce1-52cb-4d88-bb02-c513dec75d0e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.324104 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.324008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e03ce1-52cb-4d88-bb02-c513dec75d0e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.324267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.324127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g2t\" (UniqueName: \"kubernetes.io/projected/11e03ce1-52cb-4d88-bb02-c513dec75d0e-kube-api-access-r7g2t\") pod \"payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.324758 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.324738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e03ce1-52cb-4d88-bb02-c513dec75d0e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.333413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.333387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g2t\" (UniqueName: \"kubernetes.io/projected/11e03ce1-52cb-4d88-bb02-c513dec75d0e-kube-api-access-r7g2t\") pod \"payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.467208 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.467172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:03.740927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:03.740890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"]
Sep 07 11:47:03.742067 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:47:03.742042    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e03ce1_52cb_4d88_bb02_c513dec75d0e.slice/crio-5e171144fcc948f9ec1be76f189e2b91aba0ce40fde8d94efdebe7abc68d0c10 WatchSource:0}: Error finding container 5e171144fcc948f9ec1be76f189e2b91aba0ce40fde8d94efdebe7abc68d0c10: Status 404 returned error can't find the container with id 5e171144fcc948f9ec1be76f189e2b91aba0ce40fde8d94efdebe7abc68d0c10
Sep 07 11:47:04.191653 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:04.191619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh" event={"ID":"11e03ce1-52cb-4d88-bb02-c513dec75d0e","Type":"ContainerStarted","Data":"1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5"}
Sep 07 11:47:04.191836 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:04.191679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh" event={"ID":"11e03ce1-52cb-4d88-bb02-c513dec75d0e","Type":"ContainerStarted","Data":"5e171144fcc948f9ec1be76f189e2b91aba0ce40fde8d94efdebe7abc68d0c10"}
Sep 07 11:47:04.191836 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:04.191707    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:04.210318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:04.210271    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh" podStartSLOduration=1.210255093 podStartE2EDuration="1.210255093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:47:04.208150122 +0000 UTC m=+1455.682998292" watchObservedRunningTime="2026-09-07 11:47:04.210255093 +0000 UTC m=+1455.685103243"
Sep 07 11:47:07.218452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:07.218313    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:08.010937 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:08.010905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:47:08.010937 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:08.010948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:47:08.023575 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:08.023548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:47:08.225844 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:08.225813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:47:13.102324 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:13.102280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:15.199884 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:15.199853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:47:18.694615 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:18.694572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:47:20.042805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:20.042774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:21.313085 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:21.313052    2578 generic.go:356] "Generic (PLEG): container finished" podID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerID="6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd" exitCode=0
Sep 07 11:47:21.313442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:21.313129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" event={"ID":"6e56eaae-4ca1-4cba-90fc-8080ce078127","Type":"ContainerDied","Data":"6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd"}
Sep 07 11:47:21.313577 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:21.313552    2578 scope.go:117] "RemoveContainer" containerID="6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd"
Sep 07 11:47:21.314696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:21.314669    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:47:21.314795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:21.314735    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:47:21.314795 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:21.314766    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:47:22.321556 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:22.321521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" event={"ID":"6e56eaae-4ca1-4cba-90fc-8080ce078127","Type":"ContainerStarted","Data":"b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c"}
Sep 07 11:47:22.951994 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:22.951956    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:47:22.952224 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:22.952200    2578 patch_prober.go:28] interesting pod/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 07 11:47:22.952271 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:22.952250    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 07 11:47:23.397339 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:23.397304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:23.951968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:23.951934    2578 patch_prober.go:28] interesting pod/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 07 11:47:23.952138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:23.951986    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 07 11:47:24.952635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:24.952597    2578 patch_prober.go:28] interesting pod/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 07 11:47:24.953070 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:24.952649    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 07 11:47:25.952918 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:25.952884    2578 patch_prober.go:28] interesting pod/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 07 11:47:25.953291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:25.952949    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 07 11:47:26.295799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:26.295682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:26.952865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:26.952827    2578 patch_prober.go:28] interesting pod/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 07 11:47:26.953047 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:26.952891    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 07 11:47:27.956833 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:27.956804    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:47:27.957279 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:27.957261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:47:27.957889 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:27.957873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:47:29.465768 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.465724    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"]
Sep 07 11:47:29.468405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.468382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"
Sep 07 11:47:29.470719 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.470700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:47:29.470824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.470705    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:47:29.477116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.477088    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"]
Sep 07 11:47:29.577989 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.577958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffc9567-739e-41ea-a1f2-00003e12a760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn\" (UID: \"3ffc9567-739e-41ea-a1f2-00003e12a760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"
Sep 07 11:47:29.678665 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.678632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffc9567-739e-41ea-a1f2-00003e12a760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn\" (UID: \"3ffc9567-739e-41ea-a1f2-00003e12a760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"
Sep 07 11:47:29.679229 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.679209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffc9567-739e-41ea-a1f2-00003e12a760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn\" (UID: \"3ffc9567-739e-41ea-a1f2-00003e12a760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"
Sep 07 11:47:29.739987 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.739922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:29.779537 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:29.779508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"
Sep 07 11:47:30.053167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:30.053140    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"]
Sep 07 11:47:30.054601 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:47:30.054577    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffc9567_739e_41ea_a1f2_00003e12a760.slice/crio-932a54ee22a78708bb8ad01442178dc92b8a31ee0bff606217d0ba4bf498107f WatchSource:0}: Error finding container 932a54ee22a78708bb8ad01442178dc92b8a31ee0bff606217d0ba4bf498107f: Status 404 returned error can't find the container with id 932a54ee22a78708bb8ad01442178dc92b8a31ee0bff606217d0ba4bf498107f
Sep 07 11:47:30.373260 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:30.373223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn" event={"ID":"3ffc9567-739e-41ea-a1f2-00003e12a760","Type":"ContainerStarted","Data":"932a54ee22a78708bb8ad01442178dc92b8a31ee0bff606217d0ba4bf498107f"}
Sep 07 11:47:30.911678 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:47:30.911644    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffc9567_739e_41ea_a1f2_00003e12a760.slice/crio-b56a6e54b55b0dc106d86583bd8ec48b91a44d0568f8cc2430ef04f88b5bacce.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffc9567_739e_41ea_a1f2_00003e12a760.slice/crio-conmon-b56a6e54b55b0dc106d86583bd8ec48b91a44d0568f8cc2430ef04f88b5bacce.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 11:47:31.380554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:31.380522    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ffc9567-739e-41ea-a1f2-00003e12a760" containerID="b56a6e54b55b0dc106d86583bd8ec48b91a44d0568f8cc2430ef04f88b5bacce" exitCode=0
Sep 07 11:47:31.380748 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:31.380615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn" event={"ID":"3ffc9567-739e-41ea-a1f2-00003e12a760","Type":"ContainerDied","Data":"b56a6e54b55b0dc106d86583bd8ec48b91a44d0568f8cc2430ef04f88b5bacce"}
Sep 07 11:47:32.521679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:32.521655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"
Sep 07 11:47:32.606792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:32.606762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffc9567-739e-41ea-a1f2-00003e12a760-maas-api-service-ca\") pod \"3ffc9567-739e-41ea-a1f2-00003e12a760\" (UID: \"3ffc9567-739e-41ea-a1f2-00003e12a760\") "
Sep 07 11:47:32.607097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:32.607075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffc9567-739e-41ea-a1f2-00003e12a760-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ffc9567-739e-41ea-a1f2-00003e12a760" (UID: "3ffc9567-739e-41ea-a1f2-00003e12a760"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:47:32.708162 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:32.708099    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffc9567-739e-41ea-a1f2-00003e12a760-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:47:33.394611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:33.394579    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"
Sep 07 11:47:33.394773 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:33.394616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn" event={"ID":"3ffc9567-739e-41ea-a1f2-00003e12a760","Type":"ContainerDied","Data":"932a54ee22a78708bb8ad01442178dc92b8a31ee0bff606217d0ba4bf498107f"}
Sep 07 11:47:33.394773 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:33.394664    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="932a54ee22a78708bb8ad01442178dc92b8a31ee0bff606217d0ba4bf498107f"
Sep 07 11:47:36.128266 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:36.128218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:39.201188 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:39.201151    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:47:39.626868 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:39.626830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:45.216003 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.215960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:45.252069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.252037    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"]
Sep 07 11:47:45.252299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.252250    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97" podUID="98d6e7ee-3311-4385-9b1a-b5b0f407c52e" containerName="payload-pre-processing" containerID="cri-o://e10a5bf822d139fe2f2f66a626c870876af0c80ac65ed7d9b8749668f528ed78" gracePeriod=30
Sep 07 11:47:45.319692 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.319656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:45.416801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.416698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:45.418316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.418282    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"]
Sep 07 11:47:45.420850 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.420825    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cfab4-6xngn"]
Sep 07 11:47:45.470687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.470610    2578 generic.go:356] "Generic (PLEG): container finished" podID="98d6e7ee-3311-4385-9b1a-b5b0f407c52e" containerID="e10a5bf822d139fe2f2f66a626c870876af0c80ac65ed7d9b8749668f528ed78" exitCode=0
Sep 07 11:47:45.470815 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.470685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97" event={"ID":"98d6e7ee-3311-4385-9b1a-b5b0f407c52e","Type":"ContainerDied","Data":"e10a5bf822d139fe2f2f66a626c870876af0c80ac65ed7d9b8749668f528ed78"}
Sep 07 11:47:45.477748 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.477709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:45.513740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.513721    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:47:45.623705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.623673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvphw\" (UniqueName: \"kubernetes.io/projected/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-kube-api-access-wvphw\") pod \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") "
Sep 07 11:47:45.623873 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.623729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-plugins-config-volume\") pod \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\" (UID: \"98d6e7ee-3311-4385-9b1a-b5b0f407c52e\") "
Sep 07 11:47:45.624145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.624124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98d6e7ee-3311-4385-9b1a-b5b0f407c52e" (UID: "98d6e7ee-3311-4385-9b1a-b5b0f407c52e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:47:45.625738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.625715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-kube-api-access-wvphw" (OuterVolumeSpecName: "kube-api-access-wvphw") pod "98d6e7ee-3311-4385-9b1a-b5b0f407c52e" (UID: "98d6e7ee-3311-4385-9b1a-b5b0f407c52e"). InnerVolumeSpecName "kube-api-access-wvphw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:47:45.725323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.725260    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvphw\" (UniqueName: \"kubernetes.io/projected/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-kube-api-access-wvphw\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:47:45.725323 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.725284    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d6e7ee-3311-4385-9b1a-b5b0f407c52e-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:47:45.810458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:45.810425    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:46.478289 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:46.478254    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"
Sep 07 11:47:46.478754 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:46.478252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97" event={"ID":"98d6e7ee-3311-4385-9b1a-b5b0f407c52e","Type":"ContainerDied","Data":"7b300b47038b9a70515eaf75fa60a1cfa2a481bf3ba2735cd29c5678336ea841"}
Sep 07 11:47:46.478754 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:46.478412    2578 scope.go:117] "RemoveContainer" containerID="e10a5bf822d139fe2f2f66a626c870876af0c80ac65ed7d9b8749668f528ed78"
Sep 07 11:47:46.500743 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:46.500722    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"]
Sep 07 11:47:46.504158 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:46.504135    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2cfab4-6fbf5cc9f8-tbg97"]
Sep 07 11:47:46.960756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:46.960728    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ffc9567-739e-41ea-a1f2-00003e12a760" path="/var/lib/kubelet/pods/3ffc9567-739e-41ea-a1f2-00003e12a760/volumes"
Sep 07 11:47:46.961098 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:46.961084    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d6e7ee-3311-4385-9b1a-b5b0f407c52e" path="/var/lib/kubelet/pods/98d6e7ee-3311-4385-9b1a-b5b0f407c52e/volumes"
Sep 07 11:47:49.292898 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:49.292865    2578 scope.go:117] "RemoveContainer" containerID="bf4c89c28e2d7a7feca2cd4b9f99be6e37a9af02c0501b556bee6f2aed8929a7"
Sep 07 11:47:49.437386 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:49.437360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:47:49.442061 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:49.442038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:47:49.444335 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:49.444313    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:47:49.448831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:49.448814    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:47:51.844922 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:51.844886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:51.944725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:51.944671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:52.148203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.148152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:52.584750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.584673    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"]
Sep 07 11:47:52.585092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.585079    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d6e7ee-3311-4385-9b1a-b5b0f407c52e" containerName="payload-pre-processing"
Sep 07 11:47:52.585137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.585093    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d6e7ee-3311-4385-9b1a-b5b0f407c52e" containerName="payload-pre-processing"
Sep 07 11:47:52.585137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.585103    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffc9567-739e-41ea-a1f2-00003e12a760" containerName="revoke-keys"
Sep 07 11:47:52.585137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.585108    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffc9567-739e-41ea-a1f2-00003e12a760" containerName="revoke-keys"
Sep 07 11:47:52.585238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.585172    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffc9567-739e-41ea-a1f2-00003e12a760" containerName="revoke-keys"
Sep 07 11:47:52.585238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.585183    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d6e7ee-3311-4385-9b1a-b5b0f407c52e" containerName="payload-pre-processing"
Sep 07 11:47:52.588130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.588114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"
Sep 07 11:47:52.590490 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.590452    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:47:52.590604 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.590590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:47:52.597180 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.597155    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"]
Sep 07 11:47:52.686669 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.686644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce8383-f655-40d1-b3af-b81d1ef43054-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q\" (UID: \"e5ce8383-f655-40d1-b3af-b81d1ef43054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"
Sep 07 11:47:52.787229 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.787192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce8383-f655-40d1-b3af-b81d1ef43054-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q\" (UID: \"e5ce8383-f655-40d1-b3af-b81d1ef43054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"
Sep 07 11:47:52.787797 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.787779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce8383-f655-40d1-b3af-b81d1ef43054-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q\" (UID: \"e5ce8383-f655-40d1-b3af-b81d1ef43054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"
Sep 07 11:47:52.840497 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.840400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:47:52.900481 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:52.900438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"
Sep 07 11:47:53.174553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:53.174523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"]
Sep 07 11:47:53.175682 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:47:53.175652    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ce8383_f655_40d1_b3af_b81d1ef43054.slice/crio-c9337e71f198962fa6518657b8906370da5948f9afc53f00ed12406ab7afc7ba WatchSource:0}: Error finding container c9337e71f198962fa6518657b8906370da5948f9afc53f00ed12406ab7afc7ba: Status 404 returned error can't find the container with id c9337e71f198962fa6518657b8906370da5948f9afc53f00ed12406ab7afc7ba
Sep 07 11:47:53.533861 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:53.533789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q" event={"ID":"e5ce8383-f655-40d1-b3af-b81d1ef43054","Type":"ContainerStarted","Data":"816d94c2d1bc06f2b50c777786b0ded1822f83e5055263908f94266226d0478f"}
Sep 07 11:47:53.533980 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:53.533875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q" event={"ID":"e5ce8383-f655-40d1-b3af-b81d1ef43054","Type":"ContainerStarted","Data":"c9337e71f198962fa6518657b8906370da5948f9afc53f00ed12406ab7afc7ba"}
Sep 07 11:47:53.551289 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:53.551242    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q" podStartSLOduration=1.551227522 podStartE2EDuration="1.551227522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:47:53.549319294 +0000 UTC m=+1505.024167464" watchObservedRunningTime="2026-09-07 11:47:53.551227522 +0000 UTC m=+1505.026075701"
Sep 07 11:47:53.697978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:53.697949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:47:54.541081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:54.541051    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5ce8383-f655-40d1-b3af-b81d1ef43054" containerID="816d94c2d1bc06f2b50c777786b0ded1822f83e5055263908f94266226d0478f" exitCode=0
Sep 07 11:47:54.541445 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:54.541098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q" event={"ID":"e5ce8383-f655-40d1-b3af-b81d1ef43054","Type":"ContainerDied","Data":"816d94c2d1bc06f2b50c777786b0ded1822f83e5055263908f94266226d0478f"}
Sep 07 11:47:55.680784 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:55.680758    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"
Sep 07 11:47:55.817524 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:55.817422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce8383-f655-40d1-b3af-b81d1ef43054-maas-api-service-ca\") pod \"e5ce8383-f655-40d1-b3af-b81d1ef43054\" (UID: \"e5ce8383-f655-40d1-b3af-b81d1ef43054\") "
Sep 07 11:47:55.817797 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:55.817773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ce8383-f655-40d1-b3af-b81d1ef43054-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5ce8383-f655-40d1-b3af-b81d1ef43054" (UID: "e5ce8383-f655-40d1-b3af-b81d1ef43054"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:47:55.918587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:55.918561    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce8383-f655-40d1-b3af-b81d1ef43054-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:47:56.554736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:56.554698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q" event={"ID":"e5ce8383-f655-40d1-b3af-b81d1ef43054","Type":"ContainerDied","Data":"c9337e71f198962fa6518657b8906370da5948f9afc53f00ed12406ab7afc7ba"}
Sep 07 11:47:56.554736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:56.554738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"
Sep 07 11:47:56.554938 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:47:56.554758    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9337e71f198962fa6518657b8906370da5948f9afc53f00ed12406ab7afc7ba"
Sep 07 11:48:01.053869 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:01.053828    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"]
Sep 07 11:48:01.065920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:01.065867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:01.984456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:01.984420    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"]
Sep 07 11:48:02.005349 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:02.005322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:02.458825 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:02.458789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:02.801963 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:02.801876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:02.971679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:02.971652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:06.214309 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:06.214265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:06.304479 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:06.304421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:07.374693 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:07.374659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:07.475479 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:07.475436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:07.727663 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:07.727578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:07.896199 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:07.896163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:48:07.955753 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:07.955721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:08.059958 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.059882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:08.252774 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.252744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:08.280455 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.280420    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"]
Sep 07 11:48:08.280703 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.280675    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh" podUID="11e03ce1-52cb-4d88-bb02-c513dec75d0e" containerName="payload-processing" containerID="cri-o://1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5" gracePeriod=30
Sep 07 11:48:08.327566 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.325836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:08.426850 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.426821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:08.477367 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.477326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"]
Sep 07 11:48:08.480087 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.480062    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f57a16-sdg7q"]
Sep 07 11:48:08.487627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.487604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:08.536824 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.536805    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:48:08.632092 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.632068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7g2t\" (UniqueName: \"kubernetes.io/projected/11e03ce1-52cb-4d88-bb02-c513dec75d0e-kube-api-access-r7g2t\") pod \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") "
Sep 07 11:48:08.632238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.632143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e03ce1-52cb-4d88-bb02-c513dec75d0e-plugins-config-volume\") pod \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\" (UID: \"11e03ce1-52cb-4d88-bb02-c513dec75d0e\") "
Sep 07 11:48:08.632555 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.632530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e03ce1-52cb-4d88-bb02-c513dec75d0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11e03ce1-52cb-4d88-bb02-c513dec75d0e" (UID: "11e03ce1-52cb-4d88-bb02-c513dec75d0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:08.634247 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.634214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e03ce1-52cb-4d88-bb02-c513dec75d0e-kube-api-access-r7g2t" (OuterVolumeSpecName: "kube-api-access-r7g2t") pod "11e03ce1-52cb-4d88-bb02-c513dec75d0e" (UID: "11e03ce1-52cb-4d88-bb02-c513dec75d0e"). InnerVolumeSpecName "kube-api-access-r7g2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:08.640557 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.640530    2578 generic.go:356] "Generic (PLEG): container finished" podID="11e03ce1-52cb-4d88-bb02-c513dec75d0e" containerID="1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5" exitCode=0
Sep 07 11:48:08.640665 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.640586    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"
Sep 07 11:48:08.640665 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.640611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh" event={"ID":"11e03ce1-52cb-4d88-bb02-c513dec75d0e","Type":"ContainerDied","Data":"1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5"}
Sep 07 11:48:08.640756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.640664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh" event={"ID":"11e03ce1-52cb-4d88-bb02-c513dec75d0e","Type":"ContainerDied","Data":"5e171144fcc948f9ec1be76f189e2b91aba0ce40fde8d94efdebe7abc68d0c10"}
Sep 07 11:48:08.640756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.640689    2578 scope.go:117] "RemoveContainer" containerID="1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5"
Sep 07 11:48:08.651164 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.651145    2578 scope.go:117] "RemoveContainer" containerID="1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5"
Sep 07 11:48:08.651453 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:08.651434    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5\": container with ID starting with 1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5 not found: ID does not exist" containerID="1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5"
Sep 07 11:48:08.651528 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.651474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5"} err="failed to get container status \"1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5\": rpc error: code = NotFound desc = could not find container \"1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5\": container with ID starting with 1a20ac482b97591148a6eeb41bb943f2a74b55bef889b1d02a4b5353809d8bb5 not found: ID does not exist"
Sep 07 11:48:08.664370 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.664342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"]
Sep 07 11:48:08.668499 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.668478    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f57a16-8649dbf69c-g4gdh"]
Sep 07 11:48:08.733522 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.733492    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7g2t\" (UniqueName: \"kubernetes.io/projected/11e03ce1-52cb-4d88-bb02-c513dec75d0e-kube-api-access-r7g2t\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:08.733522 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.733515    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e03ce1-52cb-4d88-bb02-c513dec75d0e-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:08.888037 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.887948    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:08.960978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.960940    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e03ce1-52cb-4d88-bb02-c513dec75d0e" path="/var/lib/kubelet/pods/11e03ce1-52cb-4d88-bb02-c513dec75d0e/volumes"
Sep 07 11:48:08.961343 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:08.961323    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ce8383-f655-40d1-b3af-b81d1ef43054" path="/var/lib/kubelet/pods/e5ce8383-f655-40d1-b3af-b81d1ef43054/volumes"
Sep 07 11:48:14.663270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:14.663189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:14.764367 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:14.764332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:14.962047 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:14.961971    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"]
Sep 07 11:48:14.962236 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:14.962211    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy" containerID="cri-o://b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c" gracePeriod=30
Sep 07 11:48:14.976546 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:14.976512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:15.383101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.383069    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"]
Sep 07 11:48:15.383578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.383559    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e03ce1-52cb-4d88-bb02-c513dec75d0e" containerName="payload-processing"
Sep 07 11:48:15.383578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.383578    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e03ce1-52cb-4d88-bb02-c513dec75d0e" containerName="payload-processing"
Sep 07 11:48:15.383747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.383597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ce8383-f655-40d1-b3af-b81d1ef43054" containerName="revoke-keys"
Sep 07 11:48:15.383747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.383605    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ce8383-f655-40d1-b3af-b81d1ef43054" containerName="revoke-keys"
Sep 07 11:48:15.383747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.383718    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ce8383-f655-40d1-b3af-b81d1ef43054" containerName="revoke-keys"
Sep 07 11:48:15.383747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.383736    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e03ce1-52cb-4d88-bb02-c513dec75d0e" containerName="payload-processing"
Sep 07 11:48:15.388440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.388420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"
Sep 07 11:48:15.391006 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.390985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:48:15.391548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.391522    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:48:15.393848 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.393824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"]
Sep 07 11:48:15.400447 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.400424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989adb7d-1633-46e5-9793-7c1eb9091a93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69\" (UID: \"989adb7d-1633-46e5-9793-7c1eb9091a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"
Sep 07 11:48:15.501666 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.501632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989adb7d-1633-46e5-9793-7c1eb9091a93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69\" (UID: \"989adb7d-1633-46e5-9793-7c1eb9091a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"
Sep 07 11:48:15.502239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.502219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989adb7d-1633-46e5-9793-7c1eb9091a93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69\" (UID: \"989adb7d-1633-46e5-9793-7c1eb9091a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"
Sep 07 11:48:15.640938 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.640859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:15.702820 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.702789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"
Sep 07 11:48:15.983259 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:15.983233    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"]
Sep 07 11:48:15.985337 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:48:15.985307    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989adb7d_1633_46e5_9793_7c1eb9091a93.slice/crio-3d3a870e869c7ee713517e24c090d59b3465c3c0189d7faeaa4250bdfaa5e256 WatchSource:0}: Error finding container 3d3a870e869c7ee713517e24c090d59b3465c3c0189d7faeaa4250bdfaa5e256: Status 404 returned error can't find the container with id 3d3a870e869c7ee713517e24c090d59b3465c3c0189d7faeaa4250bdfaa5e256
Sep 07 11:48:16.073927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.073890    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg" podUID="415452db-c037-4b66-83b7-8315cfb84bbe" containerName="main" containerID="cri-o://149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4" gracePeriod=45
Sep 07 11:48:16.311247 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.311220    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:48:16.411498 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415452db-c037-4b66-83b7-8315cfb84bbe-tls-certs\") pod \"415452db-c037-4b66-83b7-8315cfb84bbe\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") "
Sep 07 11:48:16.411679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5m79\" (UniqueName: \"kubernetes.io/projected/415452db-c037-4b66-83b7-8315cfb84bbe-kube-api-access-r5m79\") pod \"415452db-c037-4b66-83b7-8315cfb84bbe\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") "
Sep 07 11:48:16.411679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-model-cache\") pod \"415452db-c037-4b66-83b7-8315cfb84bbe\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") "
Sep 07 11:48:16.411679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411604    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-tmp-dir\") pod \"415452db-c037-4b66-83b7-8315cfb84bbe\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") "
Sep 07 11:48:16.411679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-home\") pod \"415452db-c037-4b66-83b7-8315cfb84bbe\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") "
Sep 07 11:48:16.411895 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-dshm\") pod \"415452db-c037-4b66-83b7-8315cfb84bbe\" (UID: \"415452db-c037-4b66-83b7-8315cfb84bbe\") "
Sep 07 11:48:16.411895 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "415452db-c037-4b66-83b7-8315cfb84bbe" (UID: "415452db-c037-4b66-83b7-8315cfb84bbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:16.411895 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411872    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "415452db-c037-4b66-83b7-8315cfb84bbe" (UID: "415452db-c037-4b66-83b7-8315cfb84bbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:16.412023 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.411886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-home" (OuterVolumeSpecName: "home") pod "415452db-c037-4b66-83b7-8315cfb84bbe" (UID: "415452db-c037-4b66-83b7-8315cfb84bbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:16.412068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.412041    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:16.412068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.412062    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:16.412147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.412075    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:16.413711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.413688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-dshm" (OuterVolumeSpecName: "dshm") pod "415452db-c037-4b66-83b7-8315cfb84bbe" (UID: "415452db-c037-4b66-83b7-8315cfb84bbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:16.413783 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.413688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415452db-c037-4b66-83b7-8315cfb84bbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "415452db-c037-4b66-83b7-8315cfb84bbe" (UID: "415452db-c037-4b66-83b7-8315cfb84bbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:16.413783 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.413740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415452db-c037-4b66-83b7-8315cfb84bbe-kube-api-access-r5m79" (OuterVolumeSpecName: "kube-api-access-r5m79") pod "415452db-c037-4b66-83b7-8315cfb84bbe" (UID: "415452db-c037-4b66-83b7-8315cfb84bbe"). InnerVolumeSpecName "kube-api-access-r5m79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:16.512608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.512522    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415452db-c037-4b66-83b7-8315cfb84bbe-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:16.512608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.512550    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415452db-c037-4b66-83b7-8315cfb84bbe-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:16.512608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.512561    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5m79\" (UniqueName: \"kubernetes.io/projected/415452db-c037-4b66-83b7-8315cfb84bbe-kube-api-access-r5m79\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:16.696251 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.696211    2578 generic.go:356] "Generic (PLEG): container finished" podID="415452db-c037-4b66-83b7-8315cfb84bbe" containerID="149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4" exitCode=0
Sep 07 11:48:16.696436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.696283    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"
Sep 07 11:48:16.696436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.696296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg" event={"ID":"415452db-c037-4b66-83b7-8315cfb84bbe","Type":"ContainerDied","Data":"149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4"}
Sep 07 11:48:16.696436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.696354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg" event={"ID":"415452db-c037-4b66-83b7-8315cfb84bbe","Type":"ContainerDied","Data":"90078d78e6b12a0aa9b1e8cec4cc1c163c0a8fe7f6ec772da4fe7648b52a3508"}
Sep 07 11:48:16.696436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.696378    2578 scope.go:117] "RemoveContainer" containerID="149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4"
Sep 07 11:48:16.698952 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.698919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69" event={"ID":"989adb7d-1633-46e5-9793-7c1eb9091a93","Type":"ContainerStarted","Data":"a51357fbc05c634d25d6d2762cfcc810a3a3e1cc7454ba650b99aba3f7ddecbf"}
Sep 07 11:48:16.699086 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.699012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69" event={"ID":"989adb7d-1633-46e5-9793-7c1eb9091a93","Type":"ContainerStarted","Data":"3d3a870e869c7ee713517e24c090d59b3465c3c0189d7faeaa4250bdfaa5e256"}
Sep 07 11:48:16.708995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.708278    2578 scope.go:117] "RemoveContainer" containerID="149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4"
Sep 07 11:48:16.708995 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:16.708651    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4\": container with ID starting with 149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4 not found: ID does not exist" containerID="149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4"
Sep 07 11:48:16.708995 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.708684    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4"} err="failed to get container status \"149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4\": rpc error: code = NotFound desc = could not find container \"149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4\": container with ID starting with 149d7c764cb7719320968af657afc702c539eb57fe58f35d266a905e942dd9a4 not found: ID does not exist"
Sep 07 11:48:16.716016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.715970    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69" podStartSLOduration=1.7159576300000001 podStartE2EDuration="1.71595763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:48:16.714437071 +0000 UTC m=+1528.189285241" watchObservedRunningTime="2026-09-07 11:48:16.71595763 +0000 UTC m=+1528.190805778"
Sep 07 11:48:16.727606 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.727576    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"]
Sep 07 11:48:16.729961 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.729943    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b944b/rate-test-model-6b944b-kserve-6458cf9df9-r6mqg"]
Sep 07 11:48:16.960475 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:16.960441    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415452db-c037-4b66-83b7-8315cfb84bbe" path="/var/lib/kubelet/pods/415452db-c037-4b66-83b7-8315cfb84bbe/volumes"
Sep 07 11:48:17.004072 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.004036    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b" podUID="a23b25d9-2669-41d6-b616-12374e2b78b7" containerName="main" containerID="cri-o://95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba" gracePeriod=45
Sep 07 11:48:17.253137 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.253112    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:48:17.320571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320544    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-model-cache\") pod \"a23b25d9-2669-41d6-b616-12374e2b78b7\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") "
Sep 07 11:48:17.320706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-tmp-dir\") pod \"a23b25d9-2669-41d6-b616-12374e2b78b7\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") "
Sep 07 11:48:17.320706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320616    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b25d9-2669-41d6-b616-12374e2b78b7-tls-certs\") pod \"a23b25d9-2669-41d6-b616-12374e2b78b7\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") "
Sep 07 11:48:17.320706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320638    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwt5x\" (UniqueName: \"kubernetes.io/projected/a23b25d9-2669-41d6-b616-12374e2b78b7-kube-api-access-rwt5x\") pod \"a23b25d9-2669-41d6-b616-12374e2b78b7\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") "
Sep 07 11:48:17.320706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-home\") pod \"a23b25d9-2669-41d6-b616-12374e2b78b7\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") "
Sep 07 11:48:17.320944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-dshm\") pod \"a23b25d9-2669-41d6-b616-12374e2b78b7\" (UID: \"a23b25d9-2669-41d6-b616-12374e2b78b7\") "
Sep 07 11:48:17.320944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a23b25d9-2669-41d6-b616-12374e2b78b7" (UID: "a23b25d9-2669-41d6-b616-12374e2b78b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:17.320944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320850    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a23b25d9-2669-41d6-b616-12374e2b78b7" (UID: "a23b25d9-2669-41d6-b616-12374e2b78b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:17.321113 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.320949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-home" (OuterVolumeSpecName: "home") pod "a23b25d9-2669-41d6-b616-12374e2b78b7" (UID: "a23b25d9-2669-41d6-b616-12374e2b78b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:17.321153 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.321132    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:17.321185 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.321153    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:17.321185 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.321167    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:17.322839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.322809    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-dshm" (OuterVolumeSpecName: "dshm") pod "a23b25d9-2669-41d6-b616-12374e2b78b7" (UID: "a23b25d9-2669-41d6-b616-12374e2b78b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:17.322932 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.322837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23b25d9-2669-41d6-b616-12374e2b78b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a23b25d9-2669-41d6-b616-12374e2b78b7" (UID: "a23b25d9-2669-41d6-b616-12374e2b78b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:17.322932 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.322883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23b25d9-2669-41d6-b616-12374e2b78b7-kube-api-access-rwt5x" (OuterVolumeSpecName: "kube-api-access-rwt5x") pod "a23b25d9-2669-41d6-b616-12374e2b78b7" (UID: "a23b25d9-2669-41d6-b616-12374e2b78b7"). InnerVolumeSpecName "kube-api-access-rwt5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:17.422278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.422216    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b25d9-2669-41d6-b616-12374e2b78b7-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:17.422278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.422238    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b25d9-2669-41d6-b616-12374e2b78b7-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:17.422278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.422250    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwt5x\" (UniqueName: \"kubernetes.io/projected/a23b25d9-2669-41d6-b616-12374e2b78b7-kube-api-access-rwt5x\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:17.706370 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.706293    2578 generic.go:356] "Generic (PLEG): container finished" podID="989adb7d-1633-46e5-9793-7c1eb9091a93" containerID="a51357fbc05c634d25d6d2762cfcc810a3a3e1cc7454ba650b99aba3f7ddecbf" exitCode=0
Sep 07 11:48:17.706540 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.706379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69" event={"ID":"989adb7d-1633-46e5-9793-7c1eb9091a93","Type":"ContainerDied","Data":"a51357fbc05c634d25d6d2762cfcc810a3a3e1cc7454ba650b99aba3f7ddecbf"}
Sep 07 11:48:17.707688 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.707666    2578 generic.go:356] "Generic (PLEG): container finished" podID="a23b25d9-2669-41d6-b616-12374e2b78b7" containerID="95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba" exitCode=0
Sep 07 11:48:17.707770 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.707705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"
Sep 07 11:48:17.707770 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.707747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b" event={"ID":"a23b25d9-2669-41d6-b616-12374e2b78b7","Type":"ContainerDied","Data":"95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba"}
Sep 07 11:48:17.707853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.707780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b" event={"ID":"a23b25d9-2669-41d6-b616-12374e2b78b7","Type":"ContainerDied","Data":"74dd8b5536b6c2bf6d1d25b16f2a213b53d3cb96c00a52a3791df63824c7ee65"}
Sep 07 11:48:17.707853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.707802    2578 scope.go:117] "RemoveContainer" containerID="95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba"
Sep 07 11:48:17.717205 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.717188    2578 scope.go:117] "RemoveContainer" containerID="95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba"
Sep 07 11:48:17.717495 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:17.717440    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba\": container with ID starting with 95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba not found: ID does not exist" containerID="95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba"
Sep 07 11:48:17.717495 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.717482    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba"} err="failed to get container status \"95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba\": rpc error: code = NotFound desc = could not find container \"95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba\": container with ID starting with 95f4ece5bcaba26d5eae699064f17915b4aae5f181b47b81b950a154c63c25ba not found: ID does not exist"
Sep 07 11:48:17.739664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.739641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"]
Sep 07 11:48:17.747631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:17.747610    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f0a2f/rate-test-model-0f0a2f-kserve-74c69f99cf-dw26b"]
Sep 07 11:48:18.854345 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:18.854321    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"
Sep 07 11:48:18.935302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:18.935273    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989adb7d-1633-46e5-9793-7c1eb9091a93-maas-api-service-ca\") pod \"989adb7d-1633-46e5-9793-7c1eb9091a93\" (UID: \"989adb7d-1633-46e5-9793-7c1eb9091a93\") "
Sep 07 11:48:18.935623 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:18.935603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989adb7d-1633-46e5-9793-7c1eb9091a93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "989adb7d-1633-46e5-9793-7c1eb9091a93" (UID: "989adb7d-1633-46e5-9793-7c1eb9091a93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:18.961106 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:18.961079    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23b25d9-2669-41d6-b616-12374e2b78b7" path="/var/lib/kubelet/pods/a23b25d9-2669-41d6-b616-12374e2b78b7/volumes"
Sep 07 11:48:19.036344 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:19.036300    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989adb7d-1633-46e5-9793-7c1eb9091a93-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:19.724362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:19.724327    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"
Sep 07 11:48:19.724558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:19.724331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69" event={"ID":"989adb7d-1633-46e5-9793-7c1eb9091a93","Type":"ContainerDied","Data":"3d3a870e869c7ee713517e24c090d59b3465c3c0189d7faeaa4250bdfaa5e256"}
Sep 07 11:48:19.724558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:19.724452    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d3a870e869c7ee713517e24c090d59b3465c3c0189d7faeaa4250bdfaa5e256"
Sep 07 11:48:20.219678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.219655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:48:20.248285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-token\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248285 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248275    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-data\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-podinfo\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-envoy\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-certs\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248494 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-socket\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e56eaae-4ca1-4cba-90fc-8080ce078127-istiod-ca-cert\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-credential-socket\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.248705 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.248586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6gqn\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-kube-api-access-s6gqn\") pod \"6e56eaae-4ca1-4cba-90fc-8080ce078127\" (UID: \"6e56eaae-4ca1-4cba-90fc-8080ce078127\") "
Sep 07 11:48:20.251062 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.250867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:20.251273 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.251249    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:20.251693 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.251374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:20.251693 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.251480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:20.252446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.251995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e56eaae-4ca1-4cba-90fc-8080ce078127-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:20.252601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.252574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-kube-api-access-s6gqn" (OuterVolumeSpecName: "kube-api-access-s6gqn") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "kube-api-access-s6gqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:20.253783 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.253751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:20.254118 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.254089    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:48:20.254453 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.254416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e56eaae-4ca1-4cba-90fc-8080ce078127" (UID: "6e56eaae-4ca1-4cba-90fc-8080ce078127"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:20.349832 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349800    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.349832 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349827    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.349832 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349836    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.350035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349845    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e56eaae-4ca1-4cba-90fc-8080ce078127-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.350035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349854    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.350035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349862    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6gqn\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-kube-api-access-s6gqn\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.350035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349870    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.350035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349878    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.350035 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.349885    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e56eaae-4ca1-4cba-90fc-8080ce078127-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:20.732055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.732020    2578 generic.go:356] "Generic (PLEG): container finished" podID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerID="b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c" exitCode=0
Sep 07 11:48:20.732221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.732067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" event={"ID":"6e56eaae-4ca1-4cba-90fc-8080ce078127","Type":"ContainerDied","Data":"b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c"}
Sep 07 11:48:20.732221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.732082    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"
Sep 07 11:48:20.732221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.732133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq" event={"ID":"6e56eaae-4ca1-4cba-90fc-8080ce078127","Type":"ContainerDied","Data":"1373f0a7ed7cb0c71b59ffe338ca6338fb620c1179be77579d1d87b4424f05e7"}
Sep 07 11:48:20.732221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.732154    2578 scope.go:117] "RemoveContainer" containerID="b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c"
Sep 07 11:48:20.742671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.742650    2578 scope.go:117] "RemoveContainer" containerID="6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd"
Sep 07 11:48:20.751423 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.751402    2578 scope.go:117] "RemoveContainer" containerID="b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c"
Sep 07 11:48:20.751809 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:20.751753    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c\": container with ID starting with b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c not found: ID does not exist" containerID="b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c"
Sep 07 11:48:20.751944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.751813    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c"} err="failed to get container status \"b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c\": rpc error: code = NotFound desc = could not find container \"b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c\": container with ID starting with b2c5cef7f5db2d7c001f9b19ec21f92db2c453953d5363d65dfaf1061eba009c not found: ID does not exist"
Sep 07 11:48:20.751944 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.751851    2578 scope.go:117] "RemoveContainer" containerID="6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd"
Sep 07 11:48:20.752164 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:20.752138    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd\": container with ID starting with 6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd not found: ID does not exist" containerID="6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd"
Sep 07 11:48:20.752302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.752169    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd"} err="failed to get container status \"6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd\": rpc error: code = NotFound desc = could not find container \"6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd\": container with ID starting with 6d066205b64df9dd4af2288d045ab091c6640718c771e09d006b2a85722959dd not found: ID does not exist"
Sep 07 11:48:20.755010 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.754989    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"]
Sep 07 11:48:20.759931 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.759910    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f57a16-openshift-default-85d4d6f974-7bdgq"]
Sep 07 11:48:20.959450 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:20.959420    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" path="/var/lib/kubelet/pods/6e56eaae-4ca1-4cba-90fc-8080ce078127/volumes"
Sep 07 11:48:21.394515 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:21.394476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:48:25.962810 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:25.962771    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"]
Sep 07 11:48:25.963265 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:25.963079    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p" podUID="9f6a9559-7549-42a1-99ed-4e087b97f16f" containerName="payload-pre-processing" containerID="cri-o://6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a" gracePeriod=30
Sep 07 11:48:26.117290 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.117258    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"]
Sep 07 11:48:26.121458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.121433    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0c816d-psj69"]
Sep 07 11:48:26.222952 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.222928    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:48:26.301230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.301195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wknrv\" (UniqueName: \"kubernetes.io/projected/9f6a9559-7549-42a1-99ed-4e087b97f16f-kube-api-access-wknrv\") pod \"9f6a9559-7549-42a1-99ed-4e087b97f16f\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") "
Sep 07 11:48:26.301427 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.301308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6a9559-7549-42a1-99ed-4e087b97f16f-plugins-config-volume\") pod \"9f6a9559-7549-42a1-99ed-4e087b97f16f\" (UID: \"9f6a9559-7549-42a1-99ed-4e087b97f16f\") "
Sep 07 11:48:26.301738 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.301708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6a9559-7549-42a1-99ed-4e087b97f16f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f6a9559-7549-42a1-99ed-4e087b97f16f" (UID: "9f6a9559-7549-42a1-99ed-4e087b97f16f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:26.303400 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.303373    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6a9559-7549-42a1-99ed-4e087b97f16f-kube-api-access-wknrv" (OuterVolumeSpecName: "kube-api-access-wknrv") pod "9f6a9559-7549-42a1-99ed-4e087b97f16f" (UID: "9f6a9559-7549-42a1-99ed-4e087b97f16f"). InnerVolumeSpecName "kube-api-access-wknrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:26.401916 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.401885    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wknrv\" (UniqueName: \"kubernetes.io/projected/9f6a9559-7549-42a1-99ed-4e087b97f16f-kube-api-access-wknrv\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:26.401916 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.401911    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6a9559-7549-42a1-99ed-4e087b97f16f-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:26.610747 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.610717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:26.710700 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.710666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:26.775587 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.775561    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f6a9559-7549-42a1-99ed-4e087b97f16f" containerID="6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a" exitCode=0
Sep 07 11:48:26.775733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.775598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p" event={"ID":"9f6a9559-7549-42a1-99ed-4e087b97f16f","Type":"ContainerDied","Data":"6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a"}
Sep 07 11:48:26.775733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.775605    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"
Sep 07 11:48:26.775733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.775640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p" event={"ID":"9f6a9559-7549-42a1-99ed-4e087b97f16f","Type":"ContainerDied","Data":"1b262b326c9a4c4b48aad73dcbe48a6fdf2c9e7948c00f71a45acfb4f9cd1263"}
Sep 07 11:48:26.775733 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.775667    2578 scope.go:117] "RemoveContainer" containerID="6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a"
Sep 07 11:48:26.785399 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.785383    2578 scope.go:117] "RemoveContainer" containerID="6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a"
Sep 07 11:48:26.785700 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:26.785681    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a\": container with ID starting with 6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a not found: ID does not exist" containerID="6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a"
Sep 07 11:48:26.785784 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.785712    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a"} err="failed to get container status \"6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a\": rpc error: code = NotFound desc = could not find container \"6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a\": container with ID starting with 6ae4346458201bdb2e80259ee470f267ae73231dae82ab639292e35152840a2a not found: ID does not exist"
Sep 07 11:48:26.797820 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.797795    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"]
Sep 07 11:48:26.800259 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.800231    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c816d-59c976db46-zr96p"]
Sep 07 11:48:26.960656 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.960587    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="989adb7d-1633-46e5-9793-7c1eb9091a93" path="/var/lib/kubelet/pods/989adb7d-1633-46e5-9793-7c1eb9091a93/volumes"
Sep 07 11:48:26.960948 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:26.960935    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6a9559-7549-42a1-99ed-4e087b97f16f" path="/var/lib/kubelet/pods/9f6a9559-7549-42a1-99ed-4e087b97f16f/volumes"
Sep 07 11:48:28.334235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:28.334203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:30.996786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:30.996749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:48:31.304760 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:31.304683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:32.041145 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.041110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:32.325939 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.325855    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"]
Sep 07 11:48:32.326397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326384    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23b25d9-2669-41d6-b616-12374e2b78b7" containerName="main"
Sep 07 11:48:32.326440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326400    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23b25d9-2669-41d6-b616-12374e2b78b7" containerName="main"
Sep 07 11:48:32.326440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326415    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy"
Sep 07 11:48:32.326440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326424    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy"
Sep 07 11:48:32.326440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989adb7d-1633-46e5-9793-7c1eb9091a93" containerName="revoke-keys"
Sep 07 11:48:32.326589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326444    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="989adb7d-1633-46e5-9793-7c1eb9091a93" containerName="revoke-keys"
Sep 07 11:48:32.326589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326488    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy"
Sep 07 11:48:32.326589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326498    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy"
Sep 07 11:48:32.326589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326518    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6a9559-7549-42a1-99ed-4e087b97f16f" containerName="payload-pre-processing"
Sep 07 11:48:32.326589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326526    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6a9559-7549-42a1-99ed-4e087b97f16f" containerName="payload-pre-processing"
Sep 07 11:48:32.326589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326536    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415452db-c037-4b66-83b7-8315cfb84bbe" containerName="main"
Sep 07 11:48:32.326589 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326544    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="415452db-c037-4b66-83b7-8315cfb84bbe" containerName="main"
Sep 07 11:48:32.326799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326623    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="989adb7d-1633-46e5-9793-7c1eb9091a93" containerName="revoke-keys"
Sep 07 11:48:32.326799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326636    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6a9559-7549-42a1-99ed-4e087b97f16f" containerName="payload-pre-processing"
Sep 07 11:48:32.326799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326647    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="415452db-c037-4b66-83b7-8315cfb84bbe" containerName="main"
Sep 07 11:48:32.326799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326657    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy"
Sep 07 11:48:32.326799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326666    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e56eaae-4ca1-4cba-90fc-8080ce078127" containerName="istio-proxy"
Sep 07 11:48:32.326799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.326675    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23b25d9-2669-41d6-b616-12374e2b78b7" containerName="main"
Sep 07 11:48:32.331712 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.331687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.336961 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.336940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f866d4-openshift-default-dockercfg-vpj29\""
Sep 07 11:48:32.344711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.344690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"]
Sep 07 11:48:32.350135 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.350111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:32.455716 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.455681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-podinfo\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.455716 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.455719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-token\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.455985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.455767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-data\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.455985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.455836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-socket\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.455985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.455915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-envoy\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.455985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.455957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istiod-ca-cert\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.456203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.455984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-credential-socket\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.456203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.456013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8bm\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-kube-api-access-2m8bm\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.456203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.456052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-certs\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.556671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-certs\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.556860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-podinfo\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.556860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-token\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.556860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-data\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.556860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-socket\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.556860 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-envoy\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istiod-ca-cert\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-credential-socket\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.556978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8bm\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-kube-api-access-2m8bm\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557288 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.557206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-socket\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557369 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.557351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-data\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557425 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.557351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-certs\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557425 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.557379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-credential-socket\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.557968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.557941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istiod-ca-cert\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.559278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.559261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-envoy\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.559363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.559324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-podinfo\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.567648 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.567601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-token\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.568612 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.568589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8bm\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-kube-api-access-2m8bm\") pod \"e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") " pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.642692 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.642671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:32.796083 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.796033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:32.919752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.919724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"]
Sep 07 11:48:32.921122 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:48:32.921091    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58bd57_120a_4d45_9c13_0d1c20b2c536.slice/crio-c65a1f91b1bf05c646180951e407fa9589ed768ab9cc191469e19d61fec7cfa1 WatchSource:0}: Error finding container c65a1f91b1bf05c646180951e407fa9589ed768ab9cc191469e19d61fec7cfa1: Status 404 returned error can't find the container with id c65a1f91b1bf05c646180951e407fa9589ed768ab9cc191469e19d61fec7cfa1
Sep 07 11:48:32.923428 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.923393    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:48:32.923501 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.923456    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:48:32.923537 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.923509    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:48:32.954010 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:32.953928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:33.055100 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.055065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:33.224752 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.224638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:33.227311 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.227282    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"]
Sep 07 11:48:33.227579 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.227535    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy" containerID="cri-o://1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53" gracePeriod=30
Sep 07 11:48:33.321409 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.321370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:33.630168 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.630134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"]
Sep 07 11:48:33.633704 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.633686    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"
Sep 07 11:48:33.636054 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.636033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:48:33.636156 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.636032    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:48:33.641923 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.641897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"]
Sep 07 11:48:33.768245 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.768210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b989814-6013-4624-9e41-723b69063ef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq\" (UID: \"2b989814-6013-4624-9e41-723b69063ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"
Sep 07 11:48:33.820732 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.820698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" event={"ID":"0d58bd57-120a-4d45-9c13-0d1c20b2c536","Type":"ContainerStarted","Data":"e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1"}
Sep 07 11:48:33.820859 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.820759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" event={"ID":"0d58bd57-120a-4d45-9c13-0d1c20b2c536","Type":"ContainerStarted","Data":"c65a1f91b1bf05c646180951e407fa9589ed768ab9cc191469e19d61fec7cfa1"}
Sep 07 11:48:33.841831 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.841784    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podStartSLOduration=1.8417689350000002 podStartE2EDuration="1.841768935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:48:33.839643347 +0000 UTC m=+1545.314491516" watchObservedRunningTime="2026-09-07 11:48:33.841768935 +0000 UTC m=+1545.316617061"
Sep 07 11:48:33.869689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.869663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b989814-6013-4624-9e41-723b69063ef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq\" (UID: \"2b989814-6013-4624-9e41-723b69063ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"
Sep 07 11:48:33.870340 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.870322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b989814-6013-4624-9e41-723b69063ef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq\" (UID: \"2b989814-6013-4624-9e41-723b69063ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"
Sep 07 11:48:33.936161 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.936090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:33.946005 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:33.945983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"
Sep 07 11:48:34.219750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.219718    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"]
Sep 07 11:48:34.222112 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:48:34.222086    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b989814_6013_4624_9e41_723b69063ef3.slice/crio-95c06d7e724338b7b332f619e96d9a2dbe4e7c6f3fce66da80794eeec4158109 WatchSource:0}: Error finding container 95c06d7e724338b7b332f619e96d9a2dbe4e7c6f3fce66da80794eeec4158109: Status 404 returned error can't find the container with id 95c06d7e724338b7b332f619e96d9a2dbe4e7c6f3fce66da80794eeec4158109
Sep 07 11:48:34.643226 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.643191    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:34.648599 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.648573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:34.828443 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.828417    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b989814-6013-4624-9e41-723b69063ef3" containerID="0da11d5319e99a6d08295e7709002447723815ab8f22bb5f9a17cbc9ee756db9" exitCode=0
Sep 07 11:48:34.828563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.828502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq" event={"ID":"2b989814-6013-4624-9e41-723b69063ef3","Type":"ContainerDied","Data":"0da11d5319e99a6d08295e7709002447723815ab8f22bb5f9a17cbc9ee756db9"}
Sep 07 11:48:34.828608 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.828561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq" event={"ID":"2b989814-6013-4624-9e41-723b69063ef3","Type":"ContainerStarted","Data":"95c06d7e724338b7b332f619e96d9a2dbe4e7c6f3fce66da80794eeec4158109"}
Sep 07 11:48:34.828830 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.828804    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:34.829675 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:34.829657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:35.796426 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:35.796393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:35.896532 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:35.896500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:35.968048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:35.968026    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"
Sep 07 11:48:36.092270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:36.092181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b989814-6013-4624-9e41-723b69063ef3-maas-api-service-ca\") pod \"2b989814-6013-4624-9e41-723b69063ef3\" (UID: \"2b989814-6013-4624-9e41-723b69063ef3\") "
Sep 07 11:48:36.092514 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:36.092488    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b989814-6013-4624-9e41-723b69063ef3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b989814-6013-4624-9e41-723b69063ef3" (UID: "2b989814-6013-4624-9e41-723b69063ef3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:36.092604 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:36.092583    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b989814-6013-4624-9e41-723b69063ef3-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:36.841801 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:36.841770    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"
Sep 07 11:48:36.842191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:36.841798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq" event={"ID":"2b989814-6013-4624-9e41-723b69063ef3","Type":"ContainerDied","Data":"95c06d7e724338b7b332f619e96d9a2dbe4e7c6f3fce66da80794eeec4158109"}
Sep 07 11:48:36.842191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:36.841843    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c06d7e724338b7b332f619e96d9a2dbe4e7c6f3fce66da80794eeec4158109"
Sep 07 11:48:37.029500 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.029450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:37.094293 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.094208    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:37.418363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.418333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:37.528456 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.528424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:37.631234 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.631201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:37.721656 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.721580    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"]
Sep 07 11:48:37.722080 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.722062    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b989814-6013-4624-9e41-723b69063ef3" containerName="revoke-keys"
Sep 07 11:48:37.722178 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.722083    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b989814-6013-4624-9e41-723b69063ef3" containerName="revoke-keys"
Sep 07 11:48:37.722235 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.722201    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b989814-6013-4624-9e41-723b69063ef3" containerName="revoke-keys"
Sep 07 11:48:37.725348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.725329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:37.727598 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.727577    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f866d4\""
Sep 07 11:48:37.727682 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.727587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f866d4-dockercfg-jw8rf\""
Sep 07 11:48:37.733814 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.733791    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"]
Sep 07 11:48:37.806687 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.806653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6a038-945d-42a1-8969-9c87c3e1b3d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:37.806851 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.806763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbxq\" (UniqueName: \"kubernetes.io/projected/cab6a038-945d-42a1-8969-9c87c3e1b3d4-kube-api-access-sjbxq\") pod \"payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:37.908336 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.908303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6a038-945d-42a1-8969-9c87c3e1b3d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:37.908736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.908483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbxq\" (UniqueName: \"kubernetes.io/projected/cab6a038-945d-42a1-8969-9c87c3e1b3d4-kube-api-access-sjbxq\") pod \"payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:37.909019 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.908999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6a038-945d-42a1-8969-9c87c3e1b3d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:37.916210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:37.916190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbxq\" (UniqueName: \"kubernetes.io/projected/cab6a038-945d-42a1-8969-9c87c3e1b3d4-kube-api-access-sjbxq\") pod \"payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:38.037828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.037763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:38.308563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.308533    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"]
Sep 07 11:48:38.326507 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:48:38.326454    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab6a038_945d_42a1_8969_9c87c3e1b3d4.slice/crio-9d245bed3d643a85559d87e5669a6865719d3c34105e3693ca5f1b584e218d3d WatchSource:0}: Error finding container 9d245bed3d643a85559d87e5669a6865719d3c34105e3693ca5f1b584e218d3d: Status 404 returned error can't find the container with id 9d245bed3d643a85559d87e5669a6865719d3c34105e3693ca5f1b584e218d3d
Sep 07 11:48:38.482817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.482793    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:48:38.614193 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614160    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp8ql\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-kube-api-access-jp8ql\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-credential-socket\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461664cb-93d3-4e8f-a937-642f085d3f4b-istiod-ca-cert\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614357 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-podinfo\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-envoy\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614386    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-socket\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-certs\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-data\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614517    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-token\") pod \"461664cb-93d3-4e8f-a937-642f085d3f4b\" (UID: \"461664cb-93d3-4e8f-a937-642f085d3f4b\") "
Sep 07 11:48:38.614535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:38.614841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:38.614841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:38.614841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:38.615000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614882    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.615000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614901    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.615000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614916    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.615000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.614955    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.615211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.615062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461664cb-93d3-4e8f-a937-642f085d3f4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:38.616749 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.616726    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:38.617141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.617113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-kube-api-access-jp8ql" (OuterVolumeSpecName: "kube-api-access-jp8ql") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "kube-api-access-jp8ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:38.617141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.617102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:48:38.617267 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.617158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "461664cb-93d3-4e8f-a937-642f085d3f4b" (UID: "461664cb-93d3-4e8f-a937-642f085d3f4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:48:38.715442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.715416    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461664cb-93d3-4e8f-a937-642f085d3f4b-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.715442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.715437    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.715442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.715445    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.715631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.715454    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.715631 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.715473    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8ql\" (UniqueName: \"kubernetes.io/projected/461664cb-93d3-4e8f-a937-642f085d3f4b-kube-api-access-jp8ql\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:38.855771 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.855738    2578 generic.go:356] "Generic (PLEG): container finished" podID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerID="1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53" exitCode=0
Sep 07 11:48:38.855926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.855793    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"
Sep 07 11:48:38.855926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.855829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" event={"ID":"461664cb-93d3-4e8f-a937-642f085d3f4b","Type":"ContainerDied","Data":"1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53"}
Sep 07 11:48:38.855926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.855919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl" event={"ID":"461664cb-93d3-4e8f-a937-642f085d3f4b","Type":"ContainerDied","Data":"741f99bb58011d8cb751f100fbb825572e08e46d40fbb4d353d4e1278fdfb742"}
Sep 07 11:48:38.856087 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.855951    2578 scope.go:117] "RemoveContainer" containerID="1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53"
Sep 07 11:48:38.857597 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.857570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw" event={"ID":"cab6a038-945d-42a1-8969-9c87c3e1b3d4","Type":"ContainerStarted","Data":"6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b"}
Sep 07 11:48:38.857700 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.857623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw" event={"ID":"cab6a038-945d-42a1-8969-9c87c3e1b3d4","Type":"ContainerStarted","Data":"9d245bed3d643a85559d87e5669a6865719d3c34105e3693ca5f1b584e218d3d"}
Sep 07 11:48:38.857700 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.857652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:38.866459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.866442    2578 scope.go:117] "RemoveContainer" containerID="235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb"
Sep 07 11:48:38.875927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.875412    2578 scope.go:117] "RemoveContainer" containerID="1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53"
Sep 07 11:48:38.875927 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:38.875734    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53\": container with ID starting with 1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53 not found: ID does not exist" containerID="1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53"
Sep 07 11:48:38.875927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.875768    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53"} err="failed to get container status \"1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53\": rpc error: code = NotFound desc = could not find container \"1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53\": container with ID starting with 1dd50f8cb1a39d8e320d92e00096956216d27761af0f57229526209eeb20cb53 not found: ID does not exist"
Sep 07 11:48:38.875927 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.875791    2578 scope.go:117] "RemoveContainer" containerID="235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb"
Sep 07 11:48:38.876198 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:38.876127    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb\": container with ID starting with 235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb not found: ID does not exist" containerID="235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb"
Sep 07 11:48:38.876198 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.876156    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb"} err="failed to get container status \"235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb\": rpc error: code = NotFound desc = could not find container \"235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb\": container with ID starting with 235979b5a32ffe938d001ed383b1f1c368d34337f658efc302052bedc58e71eb not found: ID does not exist"
Sep 07 11:48:38.877103 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.877067    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw" podStartSLOduration=1.877055699 podStartE2EDuration="1.877055699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:48:38.875322569 +0000 UTC m=+1550.350170739" watchObservedRunningTime="2026-09-07 11:48:38.877055699 +0000 UTC m=+1550.351903847"
Sep 07 11:48:38.889811 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.889790    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"]
Sep 07 11:48:38.893630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.893609    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c816d-openshift-default-69d6cf7ccd-kg4xl"]
Sep 07 11:48:38.962172 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:38.962149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" path="/var/lib/kubelet/pods/461664cb-93d3-4e8f-a937-642f085d3f4b/volumes"
Sep 07 11:48:43.760291 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:43.760250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:44.306501 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.306445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:44.315828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.315801    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"]
Sep 07 11:48:44.318302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.318275    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2195e0-29813025-g78jf"]
Sep 07 11:48:44.347808 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.347778    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"]
Sep 07 11:48:44.348052 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.348026    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6" podUID="5d0413d4-b232-4a97-b0e1-01c86aa2adb8" containerName="payload-pre-processing" containerID="cri-o://ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da" gracePeriod=30
Sep 07 11:48:44.405609 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.405577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:44.507130 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.507005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:44.509328 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.509304    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"]
Sep 07 11:48:44.512381 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.512360    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2195e0-2plwq"]
Sep 07 11:48:44.601055 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.601032    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:48:44.607156 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.607136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:44.670577 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.670539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-plugins-config-volume\") pod \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") "
Sep 07 11:48:44.670726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.670683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hktp\" (UniqueName: \"kubernetes.io/projected/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-kube-api-access-8hktp\") pod \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\" (UID: \"5d0413d4-b232-4a97-b0e1-01c86aa2adb8\") "
Sep 07 11:48:44.670957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.670927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d0413d4-b232-4a97-b0e1-01c86aa2adb8" (UID: "5d0413d4-b232-4a97-b0e1-01c86aa2adb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:44.672838 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.672815    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-kube-api-access-8hktp" (OuterVolumeSpecName: "kube-api-access-8hktp") pod "5d0413d4-b232-4a97-b0e1-01c86aa2adb8" (UID: "5d0413d4-b232-4a97-b0e1-01c86aa2adb8"). InnerVolumeSpecName "kube-api-access-8hktp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:44.772355 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.772326    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:44.772355 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.772352    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hktp\" (UniqueName: \"kubernetes.io/projected/5d0413d4-b232-4a97-b0e1-01c86aa2adb8-kube-api-access-8hktp\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:48:44.895781 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.895747    2578 generic.go:356] "Generic (PLEG): container finished" podID="5d0413d4-b232-4a97-b0e1-01c86aa2adb8" containerID="ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da" exitCode=0
Sep 07 11:48:44.895919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.895795    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"
Sep 07 11:48:44.895919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.895832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6" event={"ID":"5d0413d4-b232-4a97-b0e1-01c86aa2adb8","Type":"ContainerDied","Data":"ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da"}
Sep 07 11:48:44.895919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.895887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6" event={"ID":"5d0413d4-b232-4a97-b0e1-01c86aa2adb8","Type":"ContainerDied","Data":"863087188a52945f04077a39abc3ea9fc5f0517873fa0e90429b60723efc8391"}
Sep 07 11:48:44.895919 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.895910    2578 scope.go:117] "RemoveContainer" containerID="ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da"
Sep 07 11:48:44.900729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.900671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:44.905855 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.905836    2578 scope.go:117] "RemoveContainer" containerID="ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da"
Sep 07 11:48:44.906157 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:48:44.906137    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da\": container with ID starting with ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da not found: ID does not exist" containerID="ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da"
Sep 07 11:48:44.906255 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.906164    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da"} err="failed to get container status \"ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da\": rpc error: code = NotFound desc = could not find container \"ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da\": container with ID starting with ac0a059e3164d009e2ab8602c6c78d936a21aa41c5c5c0704aea017d8aaa36da not found: ID does not exist"
Sep 07 11:48:44.916719 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.916696    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"]
Sep 07 11:48:44.920324 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.920301    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2195e0-67bc598c8b-ns6r6"]
Sep 07 11:48:44.960873 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.960840    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b989814-6013-4624-9e41-723b69063ef3" path="/var/lib/kubelet/pods/2b989814-6013-4624-9e41-723b69063ef3/volumes"
Sep 07 11:48:44.961118 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.961102    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a0a941-54a7-49c8-9ae3-101effc73dbf" path="/var/lib/kubelet/pods/30a0a941-54a7-49c8-9ae3-101effc73dbf/volumes"
Sep 07 11:48:44.961396 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:44.961382    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d0413d4-b232-4a97-b0e1-01c86aa2adb8" path="/var/lib/kubelet/pods/5d0413d4-b232-4a97-b0e1-01c86aa2adb8/volumes"
Sep 07 11:48:45.001871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:45.001842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:48:49.319822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:49.319791    2578 scope.go:117] "RemoveContainer" containerID="57850b7715650bb57d3f19df3d36d746a810922785c3279eb18cfacf551a4de5"
Sep 07 11:48:49.329097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:49.329078    2578 scope.go:117] "RemoveContainer" containerID="c17c002a306cb6d8b9dd14938965821b1d887fb86f9482fa97650de088f083c6"
Sep 07 11:48:49.867230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:49.867198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:48:51.036169 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:51.036134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:51.136499 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:51.136444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:51.347407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:51.347328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:54.897138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:54.897106    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:48:54.968229 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:54.968191    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerID="e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1" exitCode=0
Sep 07 11:48:54.968384 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:54.968257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" event={"ID":"0d58bd57-120a-4d45-9c13-0d1c20b2c536","Type":"ContainerDied","Data":"e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1"}
Sep 07 11:48:54.968651 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:54.968633    2578 scope.go:117] "RemoveContainer" containerID="e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1"
Sep 07 11:48:54.969750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:54.969720    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:48:54.969830 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:54.969817    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:48:54.969876 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:54.969853    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:48:55.980755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:55.980062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" event={"ID":"0d58bd57-120a-4d45-9c13-0d1c20b2c536","Type":"ContainerStarted","Data":"3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0"}
Sep 07 11:48:56.643736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:56.643700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:48:56.643906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:56.643890    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 07 11:48:56.643956 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:56.643942    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 07 11:48:57.643915 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:57.643879    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 07 11:48:57.644293 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:57.643935    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 07 11:48:58.003996 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:58.003913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:48:58.643554 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:58.643520    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 07 11:48:58.643726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:58.643579    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 07 11:48:59.643262 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:59.643227    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 07 11:48:59.643680 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:48:59.643285    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 07 11:49:00.643971 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:00.643934    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 07 11:49:00.644353 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:00.643990    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 07 11:49:00.950812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:00.950720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:01.647120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:01.647087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:49:01.647517 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:01.647364    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:49:01.648082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:01.648064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:49:08.794274 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794237    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"]
Sep 07 11:49:08.794812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794792    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy"
Sep 07 11:49:08.794903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794815    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy"
Sep 07 11:49:08.794903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794829    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0413d4-b232-4a97-b0e1-01c86aa2adb8" containerName="payload-pre-processing"
Sep 07 11:49:08.794903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794837    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0413d4-b232-4a97-b0e1-01c86aa2adb8" containerName="payload-pre-processing"
Sep 07 11:49:08.795063 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794951    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy"
Sep 07 11:49:08.795063 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794965    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d0413d4-b232-4a97-b0e1-01c86aa2adb8" containerName="payload-pre-processing"
Sep 07 11:49:08.795063 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.794980    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy"
Sep 07 11:49:08.797477 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.797440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:08.800700 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.800673    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-984178\"/\"kube-root-ca.crt\""
Sep 07 11:49:08.800802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.800721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-984178\"/\"ipp-route-984178-kserve-self-signed-certs\""
Sep 07 11:49:08.800802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.800675    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-984178\"/\"default-dockercfg-c9jmh\""
Sep 07 11:49:08.800802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.800790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-984178\"/\"openshift-service-ca.crt\""
Sep 07 11:49:08.808966 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.808944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"]
Sep 07 11:49:08.823817 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.823785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:08.898316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.898293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbg49\" (UniqueName: \"kubernetes.io/projected/60de4672-3e10-4e88-9c27-47ef9014a862-kube-api-access-tbg49\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:08.898407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.898328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-home\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:08.898407 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.898369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-model-cache\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:08.898513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.898407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-dshm\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:08.898513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.898431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60de4672-3e10-4e88-9c27-47ef9014a862-tls-certs\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:08.898513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.898486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-tmp-dir\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:08.999843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.999809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-model-cache\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.999874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-dshm\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.999901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60de4672-3e10-4e88-9c27-47ef9014a862-tls-certs\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.999956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-tmp-dir\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:08.999996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbg49\" (UniqueName: \"kubernetes.io/projected/60de4672-3e10-4e88-9c27-47ef9014a862-kube-api-access-tbg49\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.000025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-home\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000313 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.000260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-model-cache\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000382 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.000324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-tmp-dir\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.000441 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.000420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-home\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.002157 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.002138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-dshm\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.002335 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.002315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60de4672-3e10-4e88-9c27-47ef9014a862-tls-certs\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.008492 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.008452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbg49\" (UniqueName: \"kubernetes.io/projected/60de4672-3e10-4e88-9c27-47ef9014a862-kube-api-access-tbg49\") pod \"ipp-route-984178-kserve-86d985d8c5-vtpbd\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") " pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.108218 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.108154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:09.382017 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:09.381984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"]
Sep 07 11:49:09.383181 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:49:09.383154    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60de4672_3e10_4e88_9c27_47ef9014a862.slice/crio-705a6ff4aabe03229b6526d20b2b64c2c7db871f6bdf07680e45cb698e0a774e WatchSource:0}: Error finding container 705a6ff4aabe03229b6526d20b2b64c2c7db871f6bdf07680e45cb698e0a774e: Status 404 returned error can't find the container with id 705a6ff4aabe03229b6526d20b2b64c2c7db871f6bdf07680e45cb698e0a774e
Sep 07 11:49:10.071302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:10.071266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd" event={"ID":"60de4672-3e10-4e88-9c27-47ef9014a862","Type":"ContainerStarted","Data":"08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf"}
Sep 07 11:49:10.071302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:10.071307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd" event={"ID":"60de4672-3e10-4e88-9c27-47ef9014a862","Type":"ContainerStarted","Data":"705a6ff4aabe03229b6526d20b2b64c2c7db871f6bdf07680e45cb698e0a774e"}
Sep 07 11:49:10.093663 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:10.093612    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd" podStartSLOduration=2.093597847 podStartE2EDuration="2.093597847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:49:10.09128134 +0000 UTC m=+1581.566129492" watchObservedRunningTime="2026-09-07 11:49:10.093597847 +0000 UTC m=+1581.568445995"
Sep 07 11:49:12.595147 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:12.595109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:12.894777 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:12.894741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:49:19.108402 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:19.108365    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:19.108847 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:19.108417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:19.125678 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:19.125642    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:19.144623 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:19.144595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:49:28.898317 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:28.898282    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:49:35.207809 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.207776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:35.543438 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.543354    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"]
Sep 07 11:49:35.543854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.543835    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy"
Sep 07 11:49:35.543947 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.543856    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="461664cb-93d3-4e8f-a937-642f085d3f4b" containerName="istio-proxy"
Sep 07 11:49:35.547244 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.547223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.549880 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.549857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-55c3cf-openshift-default-dockercfg-m95bd\""
Sep 07 11:49:35.561909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.561885    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"]
Sep 07 11:49:35.580713 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.580682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:35.626782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-podinfo\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.626892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-socket\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.626892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-credential-socket\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.626892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-token\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.626892 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlf7z\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-kube-api-access-jlf7z\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.627033 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.627033 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-data\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.627033 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.626988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-certs\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.627127 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.627032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-envoy\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728368 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-envoy\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-podinfo\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-socket\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728574 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-credential-socket\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-token\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlf7z\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-kube-api-access-jlf7z\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728706 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-data\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728904 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-certs\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.728977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.728947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-socket\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.729223 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.729202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-certs\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.729356 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.729258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-data\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.729356 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.729315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-credential-socket\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.729455 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.729351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.731085 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.731064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-podinfo\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.731202 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.731158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-envoy\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.737304 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.737281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-token\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.737392 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.737368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlf7z\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-kube-api-access-jlf7z\") pod \"e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") " pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:35.858729 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:35.858643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:36.096246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.096205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:36.139815 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.139792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"]
Sep 07 11:49:36.141844 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:49:36.141809    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c47a1ae_051d_4e36_b33e_9ebd3ef440d8.slice/crio-2587815b22ca47da495e4515067f99c320624e506a57a9d6c6acb366b6e7ac2e WatchSource:0}: Error finding container 2587815b22ca47da495e4515067f99c320624e506a57a9d6c6acb366b6e7ac2e: Status 404 returned error can't find the container with id 2587815b22ca47da495e4515067f99c320624e506a57a9d6c6acb366b6e7ac2e
Sep 07 11:49:36.144382 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.144340    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:49:36.144497 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.144431    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:49:36.144552 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.144495    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:49:36.230969 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.230939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" event={"ID":"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8","Type":"ContainerStarted","Data":"1cdfd9aee12d8c7304bfe92de1d0d8dc0afa7704045a5e88f7b00ea9bf6942e5"}
Sep 07 11:49:36.231304 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.230991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" event={"ID":"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8","Type":"ContainerStarted","Data":"2587815b22ca47da495e4515067f99c320624e506a57a9d6c6acb366b6e7ac2e"}
Sep 07 11:49:36.254131 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.254073    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podStartSLOduration=1.254055613 podStartE2EDuration="1.254055613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:49:36.252397761 +0000 UTC m=+1607.727245912" watchObservedRunningTime="2026-09-07 11:49:36.254055613 +0000 UTC m=+1607.728903762"
Sep 07 11:49:36.859184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:36.859147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:37.865074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:37.865041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:38.244816 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:38.244724    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:38.245751 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:38.245723    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:49:38.345690 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:38.345662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:38.443671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:38.443625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:39.640074 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:39.640036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:39.703852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:39.703825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:39.978261 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:39.978187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:40.226414 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.226378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:40.319968 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.319861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:40.360519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.360489    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"]
Sep 07 11:49:40.364122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.364105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.366475 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.366448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-55c3cf-dockercfg-5fsfp\""
Sep 07 11:49:40.366573 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.366554    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-55c3cf\""
Sep 07 11:49:40.373393 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.373369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"]
Sep 07 11:49:40.472320 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.472283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21204292-8ff0-4547-aecd-c513f29d6d6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.472526 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.472327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-578v8\" (UniqueName: \"kubernetes.io/projected/21204292-8ff0-4547-aecd-c513f29d6d6d-kube-api-access-578v8\") pod \"payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.573512 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.573416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21204292-8ff0-4547-aecd-c513f29d6d6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.573512 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.573450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-578v8\" (UniqueName: \"kubernetes.io/projected/21204292-8ff0-4547-aecd-c513f29d6d6d-kube-api-access-578v8\") pod \"payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.574116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.574095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21204292-8ff0-4547-aecd-c513f29d6d6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.581762 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.581730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-578v8\" (UniqueName: \"kubernetes.io/projected/21204292-8ff0-4547-aecd-c513f29d6d6d-kube-api-access-578v8\") pod \"payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.677069 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.677039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:40.948697 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:40.948672    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"]
Sep 07 11:49:40.950264 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:49:40.950235    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21204292_8ff0_4547_aecd_c513f29d6d6d.slice/crio-b198916bdfa645d77530a2f6251801ea1a7f665f25cb5d277c745658239de87d WatchSource:0}: Error finding container b198916bdfa645d77530a2f6251801ea1a7f665f25cb5d277c745658239de87d: Status 404 returned error can't find the container with id b198916bdfa645d77530a2f6251801ea1a7f665f25cb5d277c745658239de87d
Sep 07 11:49:41.264718 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:41.264628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p" event={"ID":"21204292-8ff0-4547-aecd-c513f29d6d6d","Type":"ContainerStarted","Data":"b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a"}
Sep 07 11:49:41.264718 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:41.264686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p" event={"ID":"21204292-8ff0-4547-aecd-c513f29d6d6d","Type":"ContainerStarted","Data":"b198916bdfa645d77530a2f6251801ea1a7f665f25cb5d277c745658239de87d"}
Sep 07 11:49:41.264718 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:41.264718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:41.282034 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:41.281986    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p" podStartSLOduration=1.281972765 podStartE2EDuration="1.281972765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:49:41.279820827 +0000 UTC m=+1612.754668978" watchObservedRunningTime="2026-09-07 11:49:41.281972765 +0000 UTC m=+1612.756820914"
Sep 07 11:49:48.994385 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:48.994306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:49:50.239441 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:50.239408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:50.292920 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:50.292896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:49:52.274685 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:52.274647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:49:59.382935 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:59.382899    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerID="1cdfd9aee12d8c7304bfe92de1d0d8dc0afa7704045a5e88f7b00ea9bf6942e5" exitCode=0
Sep 07 11:49:59.383424 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:59.382972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" event={"ID":"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8","Type":"ContainerDied","Data":"1cdfd9aee12d8c7304bfe92de1d0d8dc0afa7704045a5e88f7b00ea9bf6942e5"}
Sep 07 11:49:59.383521 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:59.383502    2578 scope.go:117] "RemoveContainer" containerID="1cdfd9aee12d8c7304bfe92de1d0d8dc0afa7704045a5e88f7b00ea9bf6942e5"
Sep 07 11:49:59.384611 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:59.384582    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:49:59.384695 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:59.384654    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:49:59.384695 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:49:59.384690    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236188Ki","pods":"250"}
Sep 07 11:50:00.397412 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:00.397369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" event={"ID":"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8","Type":"ContainerStarted","Data":"1f5b8e9c2616ce62d696f3f0703250748e9aba07ff73d08fd88cda4a70cd1cb8"}
Sep 07 11:50:00.859798 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:00.859706    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:50:00.859950 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:00.859918    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 07 11:50:00.859998 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:00.859963    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 07 11:50:01.859983 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:01.859950    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 07 11:50:01.860374 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:01.860008    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 07 11:50:02.859355 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:02.859319    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 07 11:50:02.859544 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:02.859385    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 07 11:50:03.394555 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:03.394509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:03.859985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:03.859894    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 07 11:50:03.859985 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:03.859950    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 07 11:50:04.859740 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:04.859702    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 07 11:50:04.860116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:04.859784    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 07 11:50:05.862825 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:05.862781    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:50:05.863213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:05.863055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:50:05.863750 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:05.863733    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:50:10.041973 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.041937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:10.070327 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.070292    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"]
Sep 07 11:50:10.070556 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.070528    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p" podUID="21204292-8ff0-4547-aecd-c513f29d6d6d" containerName="payload-processing" containerID="cri-o://b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a" gracePeriod=30
Sep 07 11:50:10.147823 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.147790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:10.261141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.261109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:10.300635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.300584    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:50:10.342552 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.342517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:10.360079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.360057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21204292-8ff0-4547-aecd-c513f29d6d6d-plugins-config-volume\") pod \"21204292-8ff0-4547-aecd-c513f29d6d6d\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") "
Sep 07 11:50:10.360206 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.360113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-578v8\" (UniqueName: \"kubernetes.io/projected/21204292-8ff0-4547-aecd-c513f29d6d6d-kube-api-access-578v8\") pod \"21204292-8ff0-4547-aecd-c513f29d6d6d\" (UID: \"21204292-8ff0-4547-aecd-c513f29d6d6d\") "
Sep 07 11:50:10.360452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.360428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21204292-8ff0-4547-aecd-c513f29d6d6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21204292-8ff0-4547-aecd-c513f29d6d6d" (UID: "21204292-8ff0-4547-aecd-c513f29d6d6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:50:10.362600 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.362576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21204292-8ff0-4547-aecd-c513f29d6d6d-kube-api-access-578v8" (OuterVolumeSpecName: "kube-api-access-578v8") pod "21204292-8ff0-4547-aecd-c513f29d6d6d" (UID: "21204292-8ff0-4547-aecd-c513f29d6d6d"). InnerVolumeSpecName "kube-api-access-578v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:50:10.408756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.408730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:10.460941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.460904    2578 generic.go:356] "Generic (PLEG): container finished" podID="21204292-8ff0-4547-aecd-c513f29d6d6d" containerID="b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a" exitCode=0
Sep 07 11:50:10.461129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.460949    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21204292-8ff0-4547-aecd-c513f29d6d6d-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:10.461129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.460976    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-578v8\" (UniqueName: \"kubernetes.io/projected/21204292-8ff0-4547-aecd-c513f29d6d6d-kube-api-access-578v8\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:10.461129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.460960    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"
Sep 07 11:50:10.461129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.460983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p" event={"ID":"21204292-8ff0-4547-aecd-c513f29d6d6d","Type":"ContainerDied","Data":"b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a"}
Sep 07 11:50:10.461129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.461036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p" event={"ID":"21204292-8ff0-4547-aecd-c513f29d6d6d","Type":"ContainerDied","Data":"b198916bdfa645d77530a2f6251801ea1a7f665f25cb5d277c745658239de87d"}
Sep 07 11:50:10.461129 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.461057    2578 scope.go:117] "RemoveContainer" containerID="b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a"
Sep 07 11:50:10.476338 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.476318    2578 scope.go:117] "RemoveContainer" containerID="b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a"
Sep 07 11:50:10.476644 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:50:10.476622    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a\": container with ID starting with b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a not found: ID does not exist" containerID="b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a"
Sep 07 11:50:10.476725 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.476652    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a"} err="failed to get container status \"b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a\": rpc error: code = NotFound desc = could not find container \"b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a\": container with ID starting with b047c16c14409dd6e8e8f7d8320824500e8d195b33736b4840331d6c22f8fb2a not found: ID does not exist"
Sep 07 11:50:10.485242 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.485215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"]
Sep 07 11:50:10.488862 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.488842    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55c3cf-596976c645-q5d5p"]
Sep 07 11:50:10.763691 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.763659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:10.959973 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:10.959938    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21204292-8ff0-4547-aecd-c513f29d6d6d" path="/var/lib/kubelet/pods/21204292-8ff0-4547-aecd-c513f29d6d6d/volumes"
Sep 07 11:50:15.994218 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:15.994178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:50:17.127050 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:17.127016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:17.228532 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:17.228495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:17.390736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:17.390654    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"]
Sep 07 11:50:17.390921 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:17.390894    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy" containerID="cri-o://1f5b8e9c2616ce62d696f3f0703250748e9aba07ff73d08fd88cda4a70cd1cb8" gracePeriod=30
Sep 07 11:50:17.411668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:17.411632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:20.865211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:20.865173    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 11:50:20.865671 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:20.865232    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 11:50:21.360099 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.360016    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"]
Sep 07 11:50:21.360408 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.360396    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21204292-8ff0-4547-aecd-c513f29d6d6d" containerName="payload-processing"
Sep 07 11:50:21.360458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.360410    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21204292-8ff0-4547-aecd-c513f29d6d6d" containerName="payload-processing"
Sep 07 11:50:21.360521 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.360496    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21204292-8ff0-4547-aecd-c513f29d6d6d" containerName="payload-processing"
Sep 07 11:50:21.363417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.363397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"
Sep 07 11:50:21.365941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.365909    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:50:21.366036 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.366022    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:50:21.385016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.384988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"]
Sep 07 11:50:21.457630 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.457601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6b848-6281-4dd6-8143-06bc87273822-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq\" (UID: \"a1a6b848-6281-4dd6-8143-06bc87273822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"
Sep 07 11:50:21.559039 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.559009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6b848-6281-4dd6-8143-06bc87273822-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq\" (UID: \"a1a6b848-6281-4dd6-8143-06bc87273822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"
Sep 07 11:50:21.559643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.559623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6b848-6281-4dd6-8143-06bc87273822-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq\" (UID: \"a1a6b848-6281-4dd6-8143-06bc87273822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"
Sep 07 11:50:21.623736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.623712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:21.674815 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.674791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"
Sep 07 11:50:21.954826 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:21.954799    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"]
Sep 07 11:50:21.956521 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:50:21.956493    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6b848_6281_4dd6_8143_06bc87273822.slice/crio-8cfd2b7edc741157c392a12bccc561f6b834f3b255fd43fb7d70bf399a07a486 WatchSource:0}: Error finding container 8cfd2b7edc741157c392a12bccc561f6b834f3b255fd43fb7d70bf399a07a486: Status 404 returned error can't find the container with id 8cfd2b7edc741157c392a12bccc561f6b834f3b255fd43fb7d70bf399a07a486
Sep 07 11:50:22.539776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.539740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq" event={"ID":"a1a6b848-6281-4dd6-8143-06bc87273822","Type":"ContainerStarted","Data":"19c859bb73e894679a5745c13c979a639b551b0e13ebdcb8a09512eed2ff0fcf"}
Sep 07 11:50:22.540181 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.539799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq" event={"ID":"a1a6b848-6281-4dd6-8143-06bc87273822","Type":"ContainerStarted","Data":"8cfd2b7edc741157c392a12bccc561f6b834f3b255fd43fb7d70bf399a07a486"}
Sep 07 11:50:22.541936 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.541899    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerID="1f5b8e9c2616ce62d696f3f0703250748e9aba07ff73d08fd88cda4a70cd1cb8" exitCode=0
Sep 07 11:50:22.542023 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.541962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" event={"ID":"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8","Type":"ContainerDied","Data":"1f5b8e9c2616ce62d696f3f0703250748e9aba07ff73d08fd88cda4a70cd1cb8"}
Sep 07 11:50:22.542023 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.542012    2578 scope.go:117] "RemoveContainer" containerID="1cdfd9aee12d8c7304bfe92de1d0d8dc0afa7704045a5e88f7b00ea9bf6942e5"
Sep 07 11:50:22.573452 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.573406    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq" podStartSLOduration=1.573389922 podStartE2EDuration="1.573389922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:50:22.571806979 +0000 UTC m=+1654.046655143" watchObservedRunningTime="2026-09-07 11:50:22.573389922 +0000 UTC m=+1654.048238071"
Sep 07 11:50:22.625203 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.625179    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:50:22.770886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.770854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-socket\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.770916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istiod-ca-cert\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.770950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlf7z\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-kube-api-access-jlf7z\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.770983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-certs\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-credential-socket\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-envoy\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-token\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771136 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:22.771346 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771140    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-data\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771346 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-podinfo\") pod \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\" (UID: \"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8\") "
Sep 07 11:50:22.771458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:22.771458 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:50:22.771628 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:22.771628 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:22.771742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771714    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.771742 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771731    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.771849 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771747    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.771849 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771762    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.771849 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.771775    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.773173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.773142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-kube-api-access-jlf7z" (OuterVolumeSpecName: "kube-api-access-jlf7z") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "kube-api-access-jlf7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:50:22.773275 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.773237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:50:22.773338 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.773320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:22.773373 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.773331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" (UID: "3c47a1ae-051d-4e36-b33e-9ebd3ef440d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:50:22.872405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.872371    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlf7z\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-kube-api-access-jlf7z\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.872405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.872399    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.872405 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.872408    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:22.872640 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:22.872416    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:23.549411 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:23.549380    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"
Sep 07 11:50:23.549887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:23.549418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7" event={"ID":"3c47a1ae-051d-4e36-b33e-9ebd3ef440d8","Type":"ContainerDied","Data":"2587815b22ca47da495e4515067f99c320624e506a57a9d6c6acb366b6e7ac2e"}
Sep 07 11:50:23.549887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:23.549491    2578 scope.go:117] "RemoveContainer" containerID="1f5b8e9c2616ce62d696f3f0703250748e9aba07ff73d08fd88cda4a70cd1cb8"
Sep 07 11:50:23.551375 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:23.551334    2578 generic.go:356] "Generic (PLEG): container finished" podID="a1a6b848-6281-4dd6-8143-06bc87273822" containerID="19c859bb73e894679a5745c13c979a639b551b0e13ebdcb8a09512eed2ff0fcf" exitCode=0
Sep 07 11:50:23.551505 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:23.551368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq" event={"ID":"a1a6b848-6281-4dd6-8143-06bc87273822","Type":"ContainerDied","Data":"19c859bb73e894679a5745c13c979a639b551b0e13ebdcb8a09512eed2ff0fcf"}
Sep 07 11:50:23.571640 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:23.571617    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"]
Sep 07 11:50:23.577619 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:23.577596    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-55c3cf-openshift-default-5c7f966c86-294f7"]
Sep 07 11:50:24.691310 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:24.691281    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"
Sep 07 11:50:24.787548 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:24.787514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6b848-6281-4dd6-8143-06bc87273822-maas-api-service-ca\") pod \"a1a6b848-6281-4dd6-8143-06bc87273822\" (UID: \"a1a6b848-6281-4dd6-8143-06bc87273822\") "
Sep 07 11:50:24.787863 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:24.787838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a6b848-6281-4dd6-8143-06bc87273822-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1a6b848-6281-4dd6-8143-06bc87273822" (UID: "a1a6b848-6281-4dd6-8143-06bc87273822"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:50:24.889025 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:24.888998    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6b848-6281-4dd6-8143-06bc87273822-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:24.959775 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:24.959736    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" path="/var/lib/kubelet/pods/3c47a1ae-051d-4e36-b33e-9ebd3ef440d8/volumes"
Sep 07 11:50:25.565348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:25.565321    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"
Sep 07 11:50:25.565551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:25.565318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq" event={"ID":"a1a6b848-6281-4dd6-8143-06bc87273822","Type":"ContainerDied","Data":"8cfd2b7edc741157c392a12bccc561f6b834f3b255fd43fb7d70bf399a07a486"}
Sep 07 11:50:25.565551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:25.565452    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cfd2b7edc741157c392a12bccc561f6b834f3b255fd43fb7d70bf399a07a486"
Sep 07 11:50:36.856981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:36.856934    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:36.926529 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:36.926491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:37.001897 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:37.001857    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:50:37.015585 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:37.015556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:37.033358 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:37.033330    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"]
Sep 07 11:50:37.038913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:37.038882    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-984178-zfdhq"]
Sep 07 11:50:37.073434 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:37.073386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:37.428787 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:37.428728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:38.961081 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:38.961038    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a6b848-6281-4dd6-8143-06bc87273822" path="/var/lib/kubelet/pods/a1a6b848-6281-4dd6-8143-06bc87273822/volumes"
Sep 07 11:50:42.411117 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:42.411075    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"]
Sep 07 11:50:42.454262 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:42.454217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:50:47.393699 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:47.393667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:50:49.409559 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:49.409530    2578 scope.go:117] "RemoveContainer" containerID="68ac0c559e53a31ba2663b0fdaeaeff0dce7b66ff6f3e5ccfa290dc9894cd16f"
Sep 07 11:50:57.430799 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.430738    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd" podUID="60de4672-3e10-4e88-9c27-47ef9014a862" containerName="main" containerID="cri-o://08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf" gracePeriod=45
Sep 07 11:50:57.684016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.683960    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:50:57.754394 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.754367    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-model-cache\") pod \"60de4672-3e10-4e88-9c27-47ef9014a862\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") "
Sep 07 11:50:57.754606 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.754454    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbg49\" (UniqueName: \"kubernetes.io/projected/60de4672-3e10-4e88-9c27-47ef9014a862-kube-api-access-tbg49\") pod \"60de4672-3e10-4e88-9c27-47ef9014a862\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") "
Sep 07 11:50:57.754606 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.754503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-dshm\") pod \"60de4672-3e10-4e88-9c27-47ef9014a862\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") "
Sep 07 11:50:57.754720 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.754642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-home\") pod \"60de4672-3e10-4e88-9c27-47ef9014a862\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") "
Sep 07 11:50:57.754720 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.754670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-model-cache" (OuterVolumeSpecName: "model-cache") pod "60de4672-3e10-4e88-9c27-47ef9014a862" (UID: "60de4672-3e10-4e88-9c27-47ef9014a862"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:57.754841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.754759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-tmp-dir\") pod \"60de4672-3e10-4e88-9c27-47ef9014a862\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") "
Sep 07 11:50:57.754841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.754803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60de4672-3e10-4e88-9c27-47ef9014a862-tls-certs\") pod \"60de4672-3e10-4e88-9c27-47ef9014a862\" (UID: \"60de4672-3e10-4e88-9c27-47ef9014a862\") "
Sep 07 11:50:57.755109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.755091    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:57.755362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.755334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-home" (OuterVolumeSpecName: "home") pod "60de4672-3e10-4e88-9c27-47ef9014a862" (UID: "60de4672-3e10-4e88-9c27-47ef9014a862"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:57.755497 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.755352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60de4672-3e10-4e88-9c27-47ef9014a862" (UID: "60de4672-3e10-4e88-9c27-47ef9014a862"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:57.757541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.757504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60de4672-3e10-4e88-9c27-47ef9014a862-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60de4672-3e10-4e88-9c27-47ef9014a862" (UID: "60de4672-3e10-4e88-9c27-47ef9014a862"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:50:57.757664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.757643    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-dshm" (OuterVolumeSpecName: "dshm") pod "60de4672-3e10-4e88-9c27-47ef9014a862" (UID: "60de4672-3e10-4e88-9c27-47ef9014a862"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:50:57.757709 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.757660    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60de4672-3e10-4e88-9c27-47ef9014a862-kube-api-access-tbg49" (OuterVolumeSpecName: "kube-api-access-tbg49") pod "60de4672-3e10-4e88-9c27-47ef9014a862" (UID: "60de4672-3e10-4e88-9c27-47ef9014a862"). InnerVolumeSpecName "kube-api-access-tbg49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:50:57.761240 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.761217    2578 generic.go:356] "Generic (PLEG): container finished" podID="60de4672-3e10-4e88-9c27-47ef9014a862" containerID="08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf" exitCode=0
Sep 07 11:50:57.761363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.761267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"
Sep 07 11:50:57.761363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.761299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd" event={"ID":"60de4672-3e10-4e88-9c27-47ef9014a862","Type":"ContainerDied","Data":"08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf"}
Sep 07 11:50:57.761363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.761340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd" event={"ID":"60de4672-3e10-4e88-9c27-47ef9014a862","Type":"ContainerDied","Data":"705a6ff4aabe03229b6526d20b2b64c2c7db871f6bdf07680e45cb698e0a774e"}
Sep 07 11:50:57.761363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.761357    2578 scope.go:117] "RemoveContainer" containerID="08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf"
Sep 07 11:50:57.777329 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.777312    2578 scope.go:117] "RemoveContainer" containerID="08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf"
Sep 07 11:50:57.777637 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:50:57.777616    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf\": container with ID starting with 08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf not found: ID does not exist" containerID="08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf"
Sep 07 11:50:57.777686 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.777646    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf"} err="failed to get container status \"08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf\": rpc error: code = NotFound desc = could not find container \"08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf\": container with ID starting with 08a0bf0d58f3942eb0fd52162da5c3b5b0accbd870700c30b801a11fefb25fbf not found: ID does not exist"
Sep 07 11:50:57.787231 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.787208    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"]
Sep 07 11:50:57.790828 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.790805    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-984178/ipp-route-984178-kserve-86d985d8c5-vtpbd"]
Sep 07 11:50:57.856238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.856212    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:57.856238 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.856234    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60de4672-3e10-4e88-9c27-47ef9014a862-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:57.856389 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.856244    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbg49\" (UniqueName: \"kubernetes.io/projected/60de4672-3e10-4e88-9c27-47ef9014a862-kube-api-access-tbg49\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:57.856389 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.856254    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:57.856389 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:57.856263    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60de4672-3e10-4e88-9c27-47ef9014a862-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:50:58.960091 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:50:58.960056    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60de4672-3e10-4e88-9c27-47ef9014a862" path="/var/lib/kubelet/pods/60de4672-3e10-4e88-9c27-47ef9014a862/volumes"
Sep 07 11:51:18.591498 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:18.591446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:20.244859 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:20.244819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:25.860225 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:25.860179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:25.960962 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:25.960923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:26.181079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.181047    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:26.591873 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.591795    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"]
Sep 07 11:51:26.592185 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592173    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60de4672-3e10-4e88-9c27-47ef9014a862" containerName="main"
Sep 07 11:51:26.592227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592187    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60de4672-3e10-4e88-9c27-47ef9014a862" containerName="main"
Sep 07 11:51:26.592227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592209    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy"
Sep 07 11:51:26.592227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592216    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy"
Sep 07 11:51:26.592227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a6b848-6281-4dd6-8143-06bc87273822" containerName="revoke-keys"
Sep 07 11:51:26.592227 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592229    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a6b848-6281-4dd6-8143-06bc87273822" containerName="revoke-keys"
Sep 07 11:51:26.592391 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592237    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy"
Sep 07 11:51:26.592391 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592242    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy"
Sep 07 11:51:26.592391 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592304    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a6b848-6281-4dd6-8143-06bc87273822" containerName="revoke-keys"
Sep 07 11:51:26.592391 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592311    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy"
Sep 07 11:51:26.592391 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592317    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60de4672-3e10-4e88-9c27-47ef9014a862" containerName="main"
Sep 07 11:51:26.592391 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.592323    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c47a1ae-051d-4e36-b33e-9ebd3ef440d8" containerName="istio-proxy"
Sep 07 11:51:26.595404 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.595389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"
Sep 07 11:51:26.598215 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.598194    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:51:26.598436 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.598420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cm8r\""
Sep 07 11:51:26.603722 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.603700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"]
Sep 07 11:51:26.704172 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.704143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:26.705909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.705884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572225a8-a1b9-48fe-b514-cccca5f69625-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn\" (UID: \"572225a8-a1b9-48fe-b514-cccca5f69625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"
Sep 07 11:51:26.806806 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.806772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572225a8-a1b9-48fe-b514-cccca5f69625-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn\" (UID: \"572225a8-a1b9-48fe-b514-cccca5f69625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"
Sep 07 11:51:26.807418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.807389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572225a8-a1b9-48fe-b514-cccca5f69625-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn\" (UID: \"572225a8-a1b9-48fe-b514-cccca5f69625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"
Sep 07 11:51:26.907646 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:26.907616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"
Sep 07 11:51:27.180679 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:27.180648    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"]
Sep 07 11:51:27.183295 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:51:27.183254    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572225a8_a1b9_48fe_b514_cccca5f69625.slice/crio-96aa6e61dc86fc46af250692c66dada947c5c87157ca1821e7edb3397adfbc2b WatchSource:0}: Error finding container 96aa6e61dc86fc46af250692c66dada947c5c87157ca1821e7edb3397adfbc2b: Status 404 returned error can't find the container with id 96aa6e61dc86fc46af250692c66dada947c5c87157ca1821e7edb3397adfbc2b
Sep 07 11:51:27.951141 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:27.951108    2578 generic.go:356] "Generic (PLEG): container finished" podID="572225a8-a1b9-48fe-b514-cccca5f69625" containerID="04b678a16cc03d67e53408a902df744a777c94eb279e38db56894c4115ed8649" exitCode=0
Sep 07 11:51:27.951530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:27.951196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn" event={"ID":"572225a8-a1b9-48fe-b514-cccca5f69625","Type":"ContainerDied","Data":"04b678a16cc03d67e53408a902df744a777c94eb279e38db56894c4115ed8649"}
Sep 07 11:51:27.951530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:27.951248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn" event={"ID":"572225a8-a1b9-48fe-b514-cccca5f69625","Type":"ContainerStarted","Data":"96aa6e61dc86fc46af250692c66dada947c5c87157ca1821e7edb3397adfbc2b"}
Sep 07 11:51:29.095943 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:29.095919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"
Sep 07 11:51:29.228294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:29.228209    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572225a8-a1b9-48fe-b514-cccca5f69625-maas-api-service-ca\") pod \"572225a8-a1b9-48fe-b514-cccca5f69625\" (UID: \"572225a8-a1b9-48fe-b514-cccca5f69625\") "
Sep 07 11:51:29.228681 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:29.228652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572225a8-a1b9-48fe-b514-cccca5f69625-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "572225a8-a1b9-48fe-b514-cccca5f69625" (UID: "572225a8-a1b9-48fe-b514-cccca5f69625"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:51:29.329377 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:29.329330    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572225a8-a1b9-48fe-b514-cccca5f69625-maas-api-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:29.964736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:29.964704    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"
Sep 07 11:51:29.964913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:29.964708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn" event={"ID":"572225a8-a1b9-48fe-b514-cccca5f69625","Type":"ContainerDied","Data":"96aa6e61dc86fc46af250692c66dada947c5c87157ca1821e7edb3397adfbc2b"}
Sep 07 11:51:29.964913 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:29.964830    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96aa6e61dc86fc46af250692c66dada947c5c87157ca1821e7edb3397adfbc2b"
Sep 07 11:51:37.277088 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.277052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:37.283254 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.283219    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"]
Sep 07 11:51:37.283553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.283505    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw" podUID="cab6a038-945d-42a1-8969-9c87c3e1b3d4" containerName="payload-pre-processing" containerID="cri-o://6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b" gracePeriod=30
Sep 07 11:51:37.377509 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.377456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:37.434756 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.434705    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"]
Sep 07 11:51:37.436978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.436955    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f866d4-hjfsn"]
Sep 07 11:51:37.516214 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.516155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:37.542843 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.542817    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:51:37.603134 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.603104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6a038-945d-42a1-8969-9c87c3e1b3d4-plugins-config-volume\") pod \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") "
Sep 07 11:51:37.603312 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.603207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjbxq\" (UniqueName: \"kubernetes.io/projected/cab6a038-945d-42a1-8969-9c87c3e1b3d4-kube-api-access-sjbxq\") pod \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\" (UID: \"cab6a038-945d-42a1-8969-9c87c3e1b3d4\") "
Sep 07 11:51:37.603514 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.603492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab6a038-945d-42a1-8969-9c87c3e1b3d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cab6a038-945d-42a1-8969-9c87c3e1b3d4" (UID: "cab6a038-945d-42a1-8969-9c87c3e1b3d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:51:37.605278 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.605253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab6a038-945d-42a1-8969-9c87c3e1b3d4-kube-api-access-sjbxq" (OuterVolumeSpecName: "kube-api-access-sjbxq") pod "cab6a038-945d-42a1-8969-9c87c3e1b3d4" (UID: "cab6a038-945d-42a1-8969-9c87c3e1b3d4"). InnerVolumeSpecName "kube-api-access-sjbxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:51:37.703921 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.703895    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjbxq\" (UniqueName: \"kubernetes.io/projected/cab6a038-945d-42a1-8969-9c87c3e1b3d4-kube-api-access-sjbxq\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:37.703921 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.703917    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6a038-945d-42a1-8969-9c87c3e1b3d4-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:37.898714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:37.898685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:38.015650 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.015619    2578 generic.go:356] "Generic (PLEG): container finished" podID="cab6a038-945d-42a1-8969-9c87c3e1b3d4" containerID="6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b" exitCode=0
Sep 07 11:51:38.015803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.015666    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"
Sep 07 11:51:38.015803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.015707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw" event={"ID":"cab6a038-945d-42a1-8969-9c87c3e1b3d4","Type":"ContainerDied","Data":"6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b"}
Sep 07 11:51:38.015803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.015762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw" event={"ID":"cab6a038-945d-42a1-8969-9c87c3e1b3d4","Type":"ContainerDied","Data":"9d245bed3d643a85559d87e5669a6865719d3c34105e3693ca5f1b584e218d3d"}
Sep 07 11:51:38.015803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.015793    2578 scope.go:117] "RemoveContainer" containerID="6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b"
Sep 07 11:51:38.025350 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.025334    2578 scope.go:117] "RemoveContainer" containerID="6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b"
Sep 07 11:51:38.025665 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:51:38.025648    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b\": container with ID starting with 6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b not found: ID does not exist" containerID="6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b"
Sep 07 11:51:38.025714 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.025672    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b"} err="failed to get container status \"6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b\": rpc error: code = NotFound desc = could not find container \"6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b\": container with ID starting with 6580405043b4e2c09d1f30940d195dcfdee895a9efedb62b08dbe9e451d1039b not found: ID does not exist"
Sep 07 11:51:38.037497 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.037456    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"]
Sep 07 11:51:38.039903 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.039883    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f866d4-56ccbf57cd-6bmkw"]
Sep 07 11:51:38.961157 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.961120    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572225a8-a1b9-48fe-b514-cccca5f69625" path="/var/lib/kubelet/pods/572225a8-a1b9-48fe-b514-cccca5f69625/volumes"
Sep 07 11:51:38.961512 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:38.961504    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab6a038-945d-42a1-8969-9c87c3e1b3d4" path="/var/lib/kubelet/pods/cab6a038-945d-42a1-8969-9c87c3e1b3d4/volumes"
Sep 07 11:51:41.603317 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:41.603285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:51:43.783856 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:43.783827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:43.854551 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:43.854516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:44.048994 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:44.048915    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"]
Sep 07 11:51:44.049354 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:44.049150    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy" containerID="cri-o://3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0" gracePeriod=30
Sep 07 11:51:44.074362 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:44.074325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:44.543649 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:44.543598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:46.650442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:46.650398    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 11:51:46.650834 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:46.650461    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 11:51:49.301605 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.301582    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:51:49.405568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405537    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-certs\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.405803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-envoy\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.405803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-podinfo\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.405803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-data\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.405803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405744    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-socket\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.405803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405770    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m8bm\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-kube-api-access-2m8bm\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.405803 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-token\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.406122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-credential-socket\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.406122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:51:49.406122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.405867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istiod-ca-cert\") pod \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\" (UID: \"0d58bd57-120a-4d45-9c13-0d1c20b2c536\") "
Sep 07 11:51:49.406122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.406067    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:51:49.406122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.406085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:51:49.406333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.406190    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.406333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.406208    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.406333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.406221    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.406333 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.406290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:51:49.406546 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.406393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:51:49.407929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.407898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:51:49.407929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.407923    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:51:49.408116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.407945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:51:49.408116 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.407989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-kube-api-access-2m8bm" (OuterVolumeSpecName: "kube-api-access-2m8bm") pod "0d58bd57-120a-4d45-9c13-0d1c20b2c536" (UID: "0d58bd57-120a-4d45-9c13-0d1c20b2c536"). InnerVolumeSpecName "kube-api-access-2m8bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:51:49.439668 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.439648    2578 scope.go:117] "RemoveContainer" containerID="6c6cff6f297c37558f7d7115affc6790f3004e42b477ec1d74db96c3ee58262c"
Sep 07 11:51:49.448395 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.448374    2578 scope.go:117] "RemoveContainer" containerID="0a67a5d9c18ebb6efb69976fa7f4f4475a0b2dbc07150fbb96ea636e95dbc383"
Sep 07 11:51:49.507518 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.507497    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.507518 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.507515    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.507635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.507524    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m8bm\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-kube-api-access-2m8bm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.507635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.507535    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.507635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.507543    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d58bd57-120a-4d45-9c13-0d1c20b2c536-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:49.507635 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:49.507551    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d58bd57-120a-4d45-9c13-0d1c20b2c536-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:51:50.097941 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.097908    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerID="3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0" exitCode=0
Sep 07 11:51:50.098109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.097955    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"
Sep 07 11:51:50.098109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.097985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" event={"ID":"0d58bd57-120a-4d45-9c13-0d1c20b2c536","Type":"ContainerDied","Data":"3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0"}
Sep 07 11:51:50.098109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.098039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn" event={"ID":"0d58bd57-120a-4d45-9c13-0d1c20b2c536","Type":"ContainerDied","Data":"c65a1f91b1bf05c646180951e407fa9589ed768ab9cc191469e19d61fec7cfa1"}
Sep 07 11:51:50.098109 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.098060    2578 scope.go:117] "RemoveContainer" containerID="3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0"
Sep 07 11:51:50.107449 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.107429    2578 scope.go:117] "RemoveContainer" containerID="e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1"
Sep 07 11:51:50.116040 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.116022    2578 scope.go:117] "RemoveContainer" containerID="3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0"
Sep 07 11:51:50.116277 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:51:50.116260    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0\": container with ID starting with 3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0 not found: ID does not exist" containerID="3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0"
Sep 07 11:51:50.116317 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.116285    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0"} err="failed to get container status \"3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0\": rpc error: code = NotFound desc = could not find container \"3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0\": container with ID starting with 3f4a08ea06ca8a7c09c919e10b0a5b5c4125094ac501c7f6a5c9df860123b5b0 not found: ID does not exist"
Sep 07 11:51:50.116317 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.116301    2578 scope.go:117] "RemoveContainer" containerID="e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1"
Sep 07 11:51:50.116535 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:51:50.116516    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1\": container with ID starting with e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1 not found: ID does not exist" containerID="e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1"
Sep 07 11:51:50.116578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.116541    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1"} err="failed to get container status \"e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1\": rpc error: code = NotFound desc = could not find container \"e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1\": container with ID starting with e035ec088d6ca96ddba7754e1f21e733acfe8398cf4400d3b3017f022b9e90c1 not found: ID does not exist"
Sep 07 11:51:50.121102 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.121077    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"]
Sep 07 11:51:50.126302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.126281    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f866d4-openshift-default-5bfbfc7b5b-wsgqn"]
Sep 07 11:51:50.960387 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:50.960353    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" path="/var/lib/kubelet/pods/0d58bd57-120a-4d45-9c13-0d1c20b2c536/volumes"
Sep 07 11:51:55.106526 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:55.106487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:55.108287 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:55.108262    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"]
Sep 07 11:51:55.112430 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:55.112406    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b944b-29813025-8qckn"]
Sep 07 11:51:55.204711 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:55.204679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:55.306221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:55.306181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:55.386177 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:55.386147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:55.790444 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:55.790364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:51:56.959266 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:56.959232    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3bdcf7-5b17-490e-a904-c287a76a236a" path="/var/lib/kubelet/pods/bd3bdcf7-5b17-490e-a904-c287a76a236a/volumes"
Sep 07 11:51:58.999450 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:51:58.999417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:52:02.118812 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:02.118773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:02.222845 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:02.222801    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:02.361167 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:02.361128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:02.813552 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:02.813519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:13.156929 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.156894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:13.173658 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.173626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"]
Sep 07 11:52:13.173884 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.173841    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb" podUID="5d8287c5-6574-4721-be96-1473fd2ef885" containerName="payload-processing" containerID="cri-o://32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed" gracePeriod=30
Sep 07 11:52:13.228417 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.228381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:13.328507 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.328459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:13.396173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.396134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:13.423266 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.423245    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:52:13.511721 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.511687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnlw5\" (UniqueName: \"kubernetes.io/projected/5d8287c5-6574-4721-be96-1473fd2ef885-kube-api-access-dnlw5\") pod \"5d8287c5-6574-4721-be96-1473fd2ef885\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") "
Sep 07 11:52:13.511883 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.511796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8287c5-6574-4721-be96-1473fd2ef885-plugins-config-volume\") pod \"5d8287c5-6574-4721-be96-1473fd2ef885\" (UID: \"5d8287c5-6574-4721-be96-1473fd2ef885\") "
Sep 07 11:52:13.512183 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.512157    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8287c5-6574-4721-be96-1473fd2ef885-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d8287c5-6574-4721-be96-1473fd2ef885" (UID: "5d8287c5-6574-4721-be96-1473fd2ef885"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:52:13.513879 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.513855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8287c5-6574-4721-be96-1473fd2ef885-kube-api-access-dnlw5" (OuterVolumeSpecName: "kube-api-access-dnlw5") pod "5d8287c5-6574-4721-be96-1473fd2ef885" (UID: "5d8287c5-6574-4721-be96-1473fd2ef885"). InnerVolumeSpecName "kube-api-access-dnlw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:52:13.613090 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.613059    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnlw5\" (UniqueName: \"kubernetes.io/projected/5d8287c5-6574-4721-be96-1473fd2ef885-kube-api-access-dnlw5\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:13.613090 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.613086    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8287c5-6574-4721-be96-1473fd2ef885-plugins-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:13.726593 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:13.726507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:14.250695 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.250665    2578 generic.go:356] "Generic (PLEG): container finished" podID="5d8287c5-6574-4721-be96-1473fd2ef885" containerID="32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed" exitCode=0
Sep 07 11:52:14.251101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.250706    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"
Sep 07 11:52:14.251101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.250755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb" event={"ID":"5d8287c5-6574-4721-be96-1473fd2ef885","Type":"ContainerDied","Data":"32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed"}
Sep 07 11:52:14.251101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.250803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb" event={"ID":"5d8287c5-6574-4721-be96-1473fd2ef885","Type":"ContainerDied","Data":"730013aa72a7d12bc8219ee6b6fac17967db94dbeb8bbfc6611e8b0f3d0e5db9"}
Sep 07 11:52:14.251101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.250829    2578 scope.go:117] "RemoveContainer" containerID="32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed"
Sep 07 11:52:14.261505 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.261485    2578 scope.go:117] "RemoveContainer" containerID="32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed"
Sep 07 11:52:14.261865 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:52:14.261827    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed\": container with ID starting with 32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed not found: ID does not exist" containerID="32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed"
Sep 07 11:52:14.261957 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.261873    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed"} err="failed to get container status \"32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed\": rpc error: code = NotFound desc = could not find container \"32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed\": container with ID starting with 32028eae217265300ca4d7610bd42ed04bca6cb6e22cbc0de80328c2764219ed not found: ID does not exist"
Sep 07 11:52:14.272961 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.272940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"]
Sep 07 11:52:14.276877 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.276852    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f0a2f-85c77f59dc-vctmb"]
Sep 07 11:52:14.959632 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:14.959602    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d8287c5-6574-4721-be96-1473fd2ef885" path="/var/lib/kubelet/pods/5d8287c5-6574-4721-be96-1473fd2ef885/volumes"
Sep 07 11:52:15.790700 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:15.790668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:52:20.074101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:20.074061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:20.140965 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:20.140933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:20.334335 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:20.334250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"]
Sep 07 11:52:20.334496 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:20.334473    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy" containerID="cri-o://f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088" gracePeriod=30
Sep 07 11:52:20.350132 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:20.350101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:25.574249 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.574225    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:52:25.722413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-istio-token\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.722413 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17562604-4439-4954-acba-2f59a9d2bd47-istio-podinfo\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.722689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17562604-4439-4954-acba-2f59a9d2bd47-istiod-ca-cert\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.722689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-envoy\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.722689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-credential-socket\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.722689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-data\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.722689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-certs\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.722689 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kl8g\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-kube-api-access-6kl8g\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.723004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-socket\") pod \"17562604-4439-4954-acba-2f59a9d2bd47\" (UID: \"17562604-4439-4954-acba-2f59a9d2bd47\") "
Sep 07 11:52:25.723004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:52:25.723004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722863    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-data" (OuterVolumeSpecName: "istio-data") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:52:25.723004 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.722990    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:52:25.723210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.723022    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.723210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.723032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17562604-4439-4954-acba-2f59a9d2bd47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:52:25.723210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.723039    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.723210 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.723075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:52:25.724757 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.724717    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:52:25.724866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.724810    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17562604-4439-4954-acba-2f59a9d2bd47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 11:52:25.724866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.724827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-istio-token" (OuterVolumeSpecName: "istio-token") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:52:25.724942 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.724883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-kube-api-access-6kl8g" (OuterVolumeSpecName: "kube-api-access-6kl8g") pod "17562604-4439-4954-acba-2f59a9d2bd47" (UID: "17562604-4439-4954-acba-2f59a9d2bd47"). InnerVolumeSpecName "kube-api-access-6kl8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:52:25.824513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.824455    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.824513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.824505    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kl8g\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-kube-api-access-6kl8g\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.824513 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.824515    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.824735 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.824525    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17562604-4439-4954-acba-2f59a9d2bd47-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.824735 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.824534    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17562604-4439-4954-acba-2f59a9d2bd47-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.824735 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.824541    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17562604-4439-4954-acba-2f59a9d2bd47-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:25.824735 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:25.824549    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17562604-4439-4954-acba-2f59a9d2bd47-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:52:26.005230 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.005152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:52:26.326992 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.326907    2578 generic.go:356] "Generic (PLEG): container finished" podID="17562604-4439-4954-acba-2f59a9d2bd47" containerID="f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088" exitCode=0
Sep 07 11:52:26.326992 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.326953    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"
Sep 07 11:52:26.327191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.326988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" event={"ID":"17562604-4439-4954-acba-2f59a9d2bd47","Type":"ContainerDied","Data":"f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088"}
Sep 07 11:52:26.327191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.327041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42" event={"ID":"17562604-4439-4954-acba-2f59a9d2bd47","Type":"ContainerDied","Data":"41723ad8f90cf523c8231bb5b756821459f8a0c7b4737e71d2df1d701af40fb3"}
Sep 07 11:52:26.327191 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.327063    2578 scope.go:117] "RemoveContainer" containerID="f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088"
Sep 07 11:52:26.336982 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.336967    2578 scope.go:117] "RemoveContainer" containerID="cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9"
Sep 07 11:52:26.345257 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.345239    2578 scope.go:117] "RemoveContainer" containerID="f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088"
Sep 07 11:52:26.345527 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:52:26.345507    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088\": container with ID starting with f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088 not found: ID does not exist" containerID="f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088"
Sep 07 11:52:26.345579 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.345536    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088"} err="failed to get container status \"f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088\": rpc error: code = NotFound desc = could not find container \"f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088\": container with ID starting with f208573b064276846930866e36d863ccfb892be39432a2b51eb86356d74ee088 not found: ID does not exist"
Sep 07 11:52:26.345579 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.345553    2578 scope.go:117] "RemoveContainer" containerID="cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9"
Sep 07 11:52:26.345831 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:52:26.345811    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9\": container with ID starting with cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9 not found: ID does not exist" containerID="cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9"
Sep 07 11:52:26.345887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.345835    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9"} err="failed to get container status \"cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9\": rpc error: code = NotFound desc = could not find container \"cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9\": container with ID starting with cb217c89ebe095ce8a85dcd55c97e3dbefddfd8e8b333537e36e075cafa355e9 not found: ID does not exist"
Sep 07 11:52:26.352565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.352543    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"]
Sep 07 11:52:26.363644 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.363623    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f0a2f-openshift-default-54b86dfc4d-hkc42"]
Sep 07 11:52:26.815166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.815134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:26.960363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:26.960332    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17562604-4439-4954-acba-2f59a9d2bd47" path="/var/lib/kubelet/pods/17562604-4439-4954-acba-2f59a9d2bd47/volumes"
Sep 07 11:52:43.442022 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:43.441936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:52:49.478639 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:49.478610    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:52:49.481337 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:49.481310    2578 scope.go:117] "RemoveContainer" containerID="4f147a0bcdd57483ca4a978809f3d3d2f9f1d5dfafaf44c21aef3bba035b034c"
Sep 07 11:52:49.483442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:49.483415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:52:49.488535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:49.488510    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:52:49.491496 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:49.491485    2578 scope.go:117] "RemoveContainer" containerID="f318ff3e26120051a1e88db26fc2228c9a96f6993c84f9b093c14ce35b0d59c0"
Sep 07 11:52:49.492907 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:49.492889    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:52:49.502853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:52:49.502833    2578 scope.go:117] "RemoveContainer" containerID="8ed91dad034a51c28f5b6abb09545f36279a5ecebe3f3d529d1f9fa2bb5770f5"
Sep 07 11:53:22.644016 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:53:22.643978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:53:46.107553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:53:46.107509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:53:47.125121 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:53:47.125086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:53:47.446270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:53:47.446186    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:53:48.953211 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:53:48.953181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:53:49.543802 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:53:49.543768    2578 scope.go:117] "RemoveContainer" containerID="b56a6e54b55b0dc106d86583bd8ec48b91a44d0568f8cc2430ef04f88b5bacce"
Sep 07 11:54:10.992090 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:10.992054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:54:17.504213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:17.504177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:54:24.707133 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:24.707099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:54:33.397575 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:33.397532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:54:47.706683 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:47.706647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:54:49.564939 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:49.564906    2578 scope.go:117] "RemoveContainer" containerID="0da11d5319e99a6d08295e7709002447723815ab8f22bb5f9a17cbc9ee756db9"
Sep 07 11:54:49.573683 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:49.573664    2578 scope.go:117] "RemoveContainer" containerID="a51357fbc05c634d25d6d2762cfcc810a3a3e1cc7454ba650b99aba3f7ddecbf"
Sep 07 11:54:49.581187 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:49.581168    2578 scope.go:117] "RemoveContainer" containerID="816d94c2d1bc06f2b50c777786b0ded1822f83e5055263908f94266226d0478f"
Sep 07 11:54:55.591879 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:54:55.591842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:55:02.797459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:55:02.797421    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:56:08.429022 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:56:08.428939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:56:08.516158 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:56:08.516123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:56:34.289894 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:56:34.289859    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:56:40.097503 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:56:40.097448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:56:49.615901 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:56:49.615868    2578 scope.go:117] "RemoveContainer" containerID="19c859bb73e894679a5745c13c979a639b551b0e13ebdcb8a09512eed2ff0fcf"
Sep 07 11:57:02.902852 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.902816    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"]
Sep 07 11:57:02.903248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903188    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy"
Sep 07 11:57:02.903248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903202    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy"
Sep 07 11:57:02.903248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903217    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy"
Sep 07 11:57:02.903248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903223    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy"
Sep 07 11:57:02.903248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903232    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8287c5-6574-4721-be96-1473fd2ef885" containerName="payload-processing"
Sep 07 11:57:02.903248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903239    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8287c5-6574-4721-be96-1473fd2ef885" containerName="payload-processing"
Sep 07 11:57:02.903248 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903251    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab6a038-945d-42a1-8969-9c87c3e1b3d4" containerName="payload-pre-processing"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903258    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab6a038-945d-42a1-8969-9c87c3e1b3d4" containerName="payload-pre-processing"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572225a8-a1b9-48fe-b514-cccca5f69625" containerName="revoke-keys"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903269    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="572225a8-a1b9-48fe-b514-cccca5f69625" containerName="revoke-keys"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903330    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab6a038-945d-42a1-8969-9c87c3e1b3d4" containerName="payload-pre-processing"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903338    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903344    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903350    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="572225a8-a1b9-48fe-b514-cccca5f69625" containerName="revoke-keys"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903357    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903364    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy"
Sep 07 11:57:02.903571 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.903369    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8287c5-6574-4721-be96-1473fd2ef885" containerName="payload-processing"
Sep 07 11:57:02.906258 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.906242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:02.908737 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.908716    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9e36ef2a-kserve-self-signed-certs\""
Sep 07 11:57:02.915068 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.915045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"]
Sep 07 11:57:02.991909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.991876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tls-certs\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:02.992073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.991914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tmp-dir\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:02.992073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.991939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hq9l\" (UniqueName: \"kubernetes.io/projected/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-kube-api-access-6hq9l\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:02.992073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.992014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-model-cache\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:02.992073 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.992070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-home\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:02.992243 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.992105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-dshm\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:02.994263 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:02.994234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:57:03.093361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tls-certs\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tmp-dir\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq9l\" (UniqueName: \"kubernetes.io/projected/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-kube-api-access-6hq9l\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-model-cache\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093629 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-home\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093792 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-dshm\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093889 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tmp-dir\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093930 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-model-cache\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.093981 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.093962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-home\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.095849 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.095824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-dshm\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.095953 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.095858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tls-certs\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.101157 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.101135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq9l\" (UniqueName: \"kubernetes.io/projected/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-kube-api-access-6hq9l\") pod \"e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") " pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.216369 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.216286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:03.486997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.486972    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"]
Sep 07 11:57:03.488797 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:57:03.488764    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78ad3d27_0ac7_42b6_b59f_04a7c6c490ad.slice/crio-f67aee0de8f036703fb3a15c3b9cc5fdb836619d760095d1c9b6e86708333f77 WatchSource:0}: Error finding container f67aee0de8f036703fb3a15c3b9cc5fdb836619d760095d1c9b6e86708333f77: Status 404 returned error can't find the container with id f67aee0de8f036703fb3a15c3b9cc5fdb836619d760095d1c9b6e86708333f77
Sep 07 11:57:03.845807 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.845707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh" event={"ID":"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad","Type":"ContainerStarted","Data":"3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59"}
Sep 07 11:57:03.845807 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.845758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh" event={"ID":"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad","Type":"ContainerStarted","Data":"f67aee0de8f036703fb3a15c3b9cc5fdb836619d760095d1c9b6e86708333f77"}
Sep 07 11:57:03.862736 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:03.862689    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh" podStartSLOduration=1.8626730390000001 podStartE2EDuration="1.862673039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:57:03.861736486 +0000 UTC m=+2055.336584613" watchObservedRunningTime="2026-09-07 11:57:03.862673039 +0000 UTC m=+2055.337521188"
Sep 07 11:57:05.289358 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:05.289324    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:57:08.390922 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.390890    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"]
Sep 07 11:57:08.391416 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.391223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy"
Sep 07 11:57:08.391416 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.391234    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17562604-4439-4954-acba-2f59a9d2bd47" containerName="istio-proxy"
Sep 07 11:57:08.391416 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.391250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy"
Sep 07 11:57:08.391416 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.391255    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d58bd57-120a-4d45-9c13-0d1c20b2c536" containerName="istio-proxy"
Sep 07 11:57:08.394389 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.394368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.396906 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.396885    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9e36ef2a-kserve-self-signed-certs\""
Sep 07 11:57:08.405082 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.405060    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"]
Sep 07 11:57:08.410749 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.410722    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:57:08.540973 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.540942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-home\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.541125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.540986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-dshm\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.541125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.541022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393c5c70-2c83-417f-9dff-f92112eb3fb7-tls-certs\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.541125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.541057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7b44\" (UniqueName: \"kubernetes.io/projected/393c5c70-2c83-417f-9dff-f92112eb3fb7-kube-api-access-s7b44\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.541125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.541084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-tmp-dir\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.541125 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.541112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-model-cache\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-home\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642321 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-dshm\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393c5c70-2c83-417f-9dff-f92112eb3fb7-tls-certs\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642553 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7b44\" (UniqueName: \"kubernetes.io/projected/393c5c70-2c83-417f-9dff-f92112eb3fb7-kube-api-access-s7b44\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642663 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-tmp-dir\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642663 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-model-cache\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642758 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-home\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.642930 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-tmp-dir\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.643030 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.642942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-model-cache\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.644584 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.644559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-dshm\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.644866 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.644848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393c5c70-2c83-417f-9dff-f92112eb3fb7-tls-certs\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.649683 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.649663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7b44\" (UniqueName: \"kubernetes.io/projected/393c5c70-2c83-417f-9dff-f92112eb3fb7-kube-api-access-s7b44\") pod \"e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") " pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.704204 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.704181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:08.975654 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:08.975629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"]
Sep 07 11:57:08.977179 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:57:08.977153    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod393c5c70_2c83_417f_9dff_f92112eb3fb7.slice/crio-f13cf2f93186437f903f4d4830d88894e776fe59002e3b09f68c93a10e9c6952 WatchSource:0}: Error finding container f13cf2f93186437f903f4d4830d88894e776fe59002e3b09f68c93a10e9c6952: Status 404 returned error can't find the container with id f13cf2f93186437f903f4d4830d88894e776fe59002e3b09f68c93a10e9c6952
Sep 07 11:57:09.880815 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:09.880779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv" event={"ID":"393c5c70-2c83-417f-9dff-f92112eb3fb7","Type":"ContainerStarted","Data":"2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6"}
Sep 07 11:57:09.880815 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:09.880820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv" event={"ID":"393c5c70-2c83-417f-9dff-f92112eb3fb7","Type":"ContainerStarted","Data":"f13cf2f93186437f903f4d4830d88894e776fe59002e3b09f68c93a10e9c6952"}
Sep 07 11:57:09.900249 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:09.900203    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv" podStartSLOduration=1.900186421 podStartE2EDuration="1.900186421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:57:09.897661835 +0000 UTC m=+2061.372509985" watchObservedRunningTime="2026-09-07 11:57:09.900186421 +0000 UTC m=+2061.375034570"
Sep 07 11:57:12.300410 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:12.300321    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:57:13.216580 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:13.216541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:13.216746 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:13.216601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:13.229224 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:13.229196    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:13.914192 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:13.914160    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:57:18.705299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:18.705263    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:18.705299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:18.705307    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:18.717830 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:18.717803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:18.943336 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:18.943302    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:57:28.563213 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:28.563180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:57:28.663348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:28.663315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:57:46.728696 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:46.728656    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"]
Sep 07 11:57:46.749694 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:46.749660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:57:48.408542 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:48.408512    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"]
Sep 07 11:57:48.435071 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:48.435028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:57:48.789928 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:48.789841    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:57:49.519997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:49.519961    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:57:49.524239 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:49.524219    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:57:49.533380 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:49.533360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:57:49.537382 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:49.537362    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-br8kb_aaabcd49-ae2b-4838-8cdf-4cf86ace72de/ovn-acl-logging/0.log"
Sep 07 11:57:49.638533 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:49.638498    2578 scope.go:117] "RemoveContainer" containerID="04b678a16cc03d67e53408a902df744a777c94eb279e38db56894c4115ed8649"
Sep 07 11:57:56.088533 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:57:56.088500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:58:01.748854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:01.748794    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh" podUID="78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" containerName="main" containerID="cri-o://3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59" gracePeriod=45
Sep 07 11:58:01.986319 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:01.986294    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:58:02.099246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-dshm\") pod \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") "
Sep 07 11:58:02.099246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-home\") pod \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") "
Sep 07 11:58:02.099246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099219    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-model-cache\") pod \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") "
Sep 07 11:58:02.099246 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tls-certs\") pod \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") "
Sep 07 11:58:02.099558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tmp-dir\") pod \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") "
Sep 07 11:58:02.099558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hq9l\" (UniqueName: \"kubernetes.io/projected/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-kube-api-access-6hq9l\") pod \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\" (UID: \"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad\") "
Sep 07 11:58:02.099558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" (UID: "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:02.099558 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-home" (OuterVolumeSpecName: "home") pod "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" (UID: "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:02.099776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099651    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" (UID: "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:02.099776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099738    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:02.099776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099750    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:02.099776 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.099759    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:02.101446 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.101422    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-dshm" (OuterVolumeSpecName: "dshm") pod "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" (UID: "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:02.101563 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.101491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" (UID: "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:58:02.101615 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.101597    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-kube-api-access-6hq9l" (OuterVolumeSpecName: "kube-api-access-6hq9l") pod "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" (UID: "78ad3d27-0ac7-42b6-b59f-04a7c6c490ad"). InnerVolumeSpecName "kube-api-access-6hq9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:58:02.169522 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.169492    2578 generic.go:356] "Generic (PLEG): container finished" podID="78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" containerID="3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59" exitCode=0
Sep 07 11:58:02.169522 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.169521    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"
Sep 07 11:58:02.169707 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.169572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh" event={"ID":"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad","Type":"ContainerDied","Data":"3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59"}
Sep 07 11:58:02.169707 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.169612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh" event={"ID":"78ad3d27-0ac7-42b6-b59f-04a7c6c490ad","Type":"ContainerDied","Data":"f67aee0de8f036703fb3a15c3b9cc5fdb836619d760095d1c9b6e86708333f77"}
Sep 07 11:58:02.169707 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.169633    2578 scope.go:117] "RemoveContainer" containerID="3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59"
Sep 07 11:58:02.182582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.182563    2578 scope.go:117] "RemoveContainer" containerID="3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59"
Sep 07 11:58:02.182875 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:58:02.182856    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59\": container with ID starting with 3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59 not found: ID does not exist" containerID="3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59"
Sep 07 11:58:02.182926 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.182883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59"} err="failed to get container status \"3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59\": rpc error: code = NotFound desc = could not find container \"3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59\": container with ID starting with 3c93973ea3f4d84ca9f6ec8b83196b62cf389a3e86cff7c427ed08c802948d59 not found: ID does not exist"
Sep 07 11:58:02.194302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.194277    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"]
Sep 07 11:58:02.198150 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.198127    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9e36ef2a-kserve-6f77d4b49f-d5lxh"]
Sep 07 11:58:02.200221 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.200207    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hq9l\" (UniqueName: \"kubernetes.io/projected/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-kube-api-access-6hq9l\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:02.200270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.200223    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:02.200270 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.200235    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:02.959982 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:02.959946    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" path="/var/lib/kubelet/pods/78ad3d27-0ac7-42b6-b59f-04a7c6c490ad/volumes"
Sep 07 11:58:03.428432 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.428365    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv" podUID="393c5c70-2c83-417f-9dff-f92112eb3fb7" containerName="main" containerID="cri-o://2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6" gracePeriod=45
Sep 07 11:58:03.683702 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.683638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:58:03.816425 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-dshm\") pod \"393c5c70-2c83-417f-9dff-f92112eb3fb7\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") "
Sep 07 11:58:03.816624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-tmp-dir\") pod \"393c5c70-2c83-417f-9dff-f92112eb3fb7\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") "
Sep 07 11:58:03.816624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-home\") pod \"393c5c70-2c83-417f-9dff-f92112eb3fb7\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") "
Sep 07 11:58:03.816624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393c5c70-2c83-417f-9dff-f92112eb3fb7-tls-certs\") pod \"393c5c70-2c83-417f-9dff-f92112eb3fb7\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") "
Sep 07 11:58:03.816624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816558    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7b44\" (UniqueName: \"kubernetes.io/projected/393c5c70-2c83-417f-9dff-f92112eb3fb7-kube-api-access-s7b44\") pod \"393c5c70-2c83-417f-9dff-f92112eb3fb7\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") "
Sep 07 11:58:03.816624 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-model-cache\") pod \"393c5c70-2c83-417f-9dff-f92112eb3fb7\" (UID: \"393c5c70-2c83-417f-9dff-f92112eb3fb7\") "
Sep 07 11:58:03.816894 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "393c5c70-2c83-417f-9dff-f92112eb3fb7" (UID: "393c5c70-2c83-417f-9dff-f92112eb3fb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:03.816894 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-home" (OuterVolumeSpecName: "home") pod "393c5c70-2c83-417f-9dff-f92112eb3fb7" (UID: "393c5c70-2c83-417f-9dff-f92112eb3fb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:03.816894 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816853    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "393c5c70-2c83-417f-9dff-f92112eb3fb7" (UID: "393c5c70-2c83-417f-9dff-f92112eb3fb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:03.816894 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.816884    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:03.818652 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.818623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-dshm" (OuterVolumeSpecName: "dshm") pod "393c5c70-2c83-417f-9dff-f92112eb3fb7" (UID: "393c5c70-2c83-417f-9dff-f92112eb3fb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:58:03.818652 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.818634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393c5c70-2c83-417f-9dff-f92112eb3fb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "393c5c70-2c83-417f-9dff-f92112eb3fb7" (UID: "393c5c70-2c83-417f-9dff-f92112eb3fb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:58:03.818797 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.818661    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393c5c70-2c83-417f-9dff-f92112eb3fb7-kube-api-access-s7b44" (OuterVolumeSpecName: "kube-api-access-s7b44") pod "393c5c70-2c83-417f-9dff-f92112eb3fb7" (UID: "393c5c70-2c83-417f-9dff-f92112eb3fb7"). InnerVolumeSpecName "kube-api-access-s7b44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:58:03.917578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.917551    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:03.917578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.917572    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:03.917578 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.917581    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393c5c70-2c83-417f-9dff-f92112eb3fb7-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:03.917781 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.917591    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7b44\" (UniqueName: \"kubernetes.io/projected/393c5c70-2c83-417f-9dff-f92112eb3fb7-kube-api-access-s7b44\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:03.917781 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:03.917600    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393c5c70-2c83-417f-9dff-f92112eb3fb7-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:04.182097 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.182066    2578 generic.go:356] "Generic (PLEG): container finished" podID="393c5c70-2c83-417f-9dff-f92112eb3fb7" containerID="2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6" exitCode=0
Sep 07 11:58:04.182536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.182109    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"
Sep 07 11:58:04.182536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.182111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv" event={"ID":"393c5c70-2c83-417f-9dff-f92112eb3fb7","Type":"ContainerDied","Data":"2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6"}
Sep 07 11:58:04.182536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.182165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv" event={"ID":"393c5c70-2c83-417f-9dff-f92112eb3fb7","Type":"ContainerDied","Data":"f13cf2f93186437f903f4d4830d88894e776fe59002e3b09f68c93a10e9c6952"}
Sep 07 11:58:04.182536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.182193    2578 scope.go:117] "RemoveContainer" containerID="2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6"
Sep 07 11:58:04.191530 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.191513    2578 scope.go:117] "RemoveContainer" containerID="2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6"
Sep 07 11:58:04.191783 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:58:04.191764    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6\": container with ID starting with 2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6 not found: ID does not exist" containerID="2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6"
Sep 07 11:58:04.191839 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.191789    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6"} err="failed to get container status \"2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6\": rpc error: code = NotFound desc = could not find container \"2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6\": container with ID starting with 2a8374f278c1842b184834027b79f2b6de4cd4caa45c41eebeaa6834f2f728f6 not found: ID does not exist"
Sep 07 11:58:04.202434 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.202411    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"]
Sep 07 11:58:04.206042 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.206021    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9e36ef2a-kserve-7b8fcc97dc-smswv"]
Sep 07 11:58:04.961002 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:04.960968    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393c5c70-2c83-417f-9dff-f92112eb3fb7" path="/var/lib/kubelet/pods/393c5c70-2c83-417f-9dff-f92112eb3fb7/volumes"
Sep 07 11:58:05.595418 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.595384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:58:05.752782 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.752750    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"]
Sep 07 11:58:05.753173 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.753152    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" containerName="main"
Sep 07 11:58:05.753256 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.753177    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" containerName="main"
Sep 07 11:58:05.753256 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.753197    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393c5c70-2c83-417f-9dff-f92112eb3fb7" containerName="main"
Sep 07 11:58:05.753256 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.753203    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="393c5c70-2c83-417f-9dff-f92112eb3fb7" containerName="main"
Sep 07 11:58:05.753363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.753280    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ad3d27-0ac7-42b6-b59f-04a7c6c490ad" containerName="main"
Sep 07 11:58:05.753363 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.753290    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="393c5c70-2c83-417f-9dff-f92112eb3fb7" containerName="main"
Sep 07 11:58:05.757615 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.757597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:05.760047 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.760026    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:58:05.760886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.760864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gtjll\""
Sep 07 11:58:05.760886 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.760879    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 11:58:05.761062 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.760887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 11:58:05.761062 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.760995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 11:58:05.767559 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.767532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"]
Sep 07 11:58:05.932484 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.932428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btxk\" (UniqueName: \"kubernetes.io/projected/bba2bf1d-61c8-4ba5-bd91-b78aff066405-kube-api-access-2btxk\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:05.932643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.932508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-metrics-certs\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:05.932643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.932590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-maas-api-tls\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:05.932643 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:05.932612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba2bf1d-61c8-4ba5-bd91-b78aff066405-openshift-service-ca\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.033633 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.033605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-metrics-certs\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.033777 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.033672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-maas-api-tls\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.033777 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.033724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba2bf1d-61c8-4ba5-bd91-b78aff066405-openshift-service-ca\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.033885 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.033795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btxk\" (UniqueName: \"kubernetes.io/projected/bba2bf1d-61c8-4ba5-bd91-b78aff066405-kube-api-access-2btxk\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.034359 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.034337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba2bf1d-61c8-4ba5-bd91-b78aff066405-openshift-service-ca\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.036048 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.036015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-metrics-certs\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.036212 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.036195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-maas-api-tls\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.041101 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.041078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btxk\" (UniqueName: \"kubernetes.io/projected/bba2bf1d-61c8-4ba5-bd91-b78aff066405-kube-api-access-2btxk\") pod \"maas-api-79f7b4565f-gxd76\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.069279 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.069255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:06.342557 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.342481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"]
Sep 07 11:58:06.345189 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:58:06.345164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba2bf1d_61c8_4ba5_bd91_b78aff066405.slice/crio-6227266d7da20c6f992b920a2ba8a4ed9e1b528df117ff8e59bb0215c2c787ca WatchSource:0}: Error finding container 6227266d7da20c6f992b920a2ba8a4ed9e1b528df117ff8e59bb0215c2c787ca: Status 404 returned error can't find the container with id 6227266d7da20c6f992b920a2ba8a4ed9e1b528df117ff8e59bb0215c2c787ca
Sep 07 11:58:06.346541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:06.346526    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:58:07.200306 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:07.200272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76" event={"ID":"bba2bf1d-61c8-4ba5-bd91-b78aff066405","Type":"ContainerStarted","Data":"6227266d7da20c6f992b920a2ba8a4ed9e1b528df117ff8e59bb0215c2c787ca"}
Sep 07 11:58:09.213841 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:09.213806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76" event={"ID":"bba2bf1d-61c8-4ba5-bd91-b78aff066405","Type":"ContainerStarted","Data":"6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04"}
Sep 07 11:58:09.214183 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:09.213947    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:09.232865 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:09.232820    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76" podStartSLOduration=1.862153315 podStartE2EDuration="4.232805442s" totalImagesPullingTime="2.370652127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:58:09.230519335 +0000 UTC m=+2120.705367508" watchObservedRunningTime="2026-09-07 11:58:09.232805442 +0000 UTC m=+2120.707653595"
Sep 07 11:58:13.397260 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:13.397228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:58:13.986627 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:13.986591    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"]
Sep 07 11:58:13.986794 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:13.986771    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76" podUID="bba2bf1d-61c8-4ba5-bd91-b78aff066405" containerName="maas-api" containerID="cri-o://6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04" gracePeriod=30
Sep 07 11:58:13.992786 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:13.992755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:14.228122 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.228096    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:14.240820 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.240749    2578 generic.go:356] "Generic (PLEG): container finished" podID="bba2bf1d-61c8-4ba5-bd91-b78aff066405" containerID="6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04" exitCode=0
Sep 07 11:58:14.240820 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.240794    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"
Sep 07 11:58:14.240820 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.240814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76" event={"ID":"bba2bf1d-61c8-4ba5-bd91-b78aff066405","Type":"ContainerDied","Data":"6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04"}
Sep 07 11:58:14.240991 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.240863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76" event={"ID":"bba2bf1d-61c8-4ba5-bd91-b78aff066405","Type":"ContainerDied","Data":"6227266d7da20c6f992b920a2ba8a4ed9e1b528df117ff8e59bb0215c2c787ca"}
Sep 07 11:58:14.240991 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.240894    2578 scope.go:117] "RemoveContainer" containerID="6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04"
Sep 07 11:58:14.251474 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.251441    2578 scope.go:117] "RemoveContainer" containerID="6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04"
Sep 07 11:58:14.251765 ip-10-0-129-163 kubenswrapper[2578]: E0907 11:58:14.251743    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04\": container with ID starting with 6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04 not found: ID does not exist" containerID="6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04"
Sep 07 11:58:14.251849 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.251790    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04"} err="failed to get container status \"6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04\": rpc error: code = NotFound desc = could not find container \"6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04\": container with ID starting with 6bfe6dc09a5e584154602b3e390988ea2f66b26b0cb30c2d6e0c489c3d9b0c04 not found: ID does not exist"
Sep 07 11:58:14.305196 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.305172    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-metrics-certs\") pod \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") "
Sep 07 11:58:14.305316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.305214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2btxk\" (UniqueName: \"kubernetes.io/projected/bba2bf1d-61c8-4ba5-bd91-b78aff066405-kube-api-access-2btxk\") pod \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") "
Sep 07 11:58:14.305316 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.305251    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba2bf1d-61c8-4ba5-bd91-b78aff066405-openshift-service-ca\") pod \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") "
Sep 07 11:58:14.305442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.305334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-maas-api-tls\") pod \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\" (UID: \"bba2bf1d-61c8-4ba5-bd91-b78aff066405\") "
Sep 07 11:58:14.305664 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.305640    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba2bf1d-61c8-4ba5-bd91-b78aff066405-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bba2bf1d-61c8-4ba5-bd91-b78aff066405" (UID: "bba2bf1d-61c8-4ba5-bd91-b78aff066405"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:58:14.307490 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.307450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bba2bf1d-61c8-4ba5-bd91-b78aff066405" (UID: "bba2bf1d-61c8-4ba5-bd91-b78aff066405"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:58:14.307490 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.307456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba2bf1d-61c8-4ba5-bd91-b78aff066405-kube-api-access-2btxk" (OuterVolumeSpecName: "kube-api-access-2btxk") pod "bba2bf1d-61c8-4ba5-bd91-b78aff066405" (UID: "bba2bf1d-61c8-4ba5-bd91-b78aff066405"). InnerVolumeSpecName "kube-api-access-2btxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:58:14.307620 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.307485    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bba2bf1d-61c8-4ba5-bd91-b78aff066405" (UID: "bba2bf1d-61c8-4ba5-bd91-b78aff066405"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:58:14.406166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.406142    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-maas-api-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:14.406166 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.406164    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba2bf1d-61c8-4ba5-bd91-b78aff066405-metrics-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:14.406515 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.406190    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btxk\" (UniqueName: \"kubernetes.io/projected/bba2bf1d-61c8-4ba5-bd91-b78aff066405-kube-api-access-2btxk\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:14.406515 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.406200    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba2bf1d-61c8-4ba5-bd91-b78aff066405-openshift-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Sep 07 11:58:14.562871 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.562841    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"]
Sep 07 11:58:14.568601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.568570    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-gxd76"]
Sep 07 11:58:14.960328 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:14.960293    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba2bf1d-61c8-4ba5-bd91-b78aff066405" path="/var/lib/kubelet/pods/bba2bf1d-61c8-4ba5-bd91-b78aff066405/volumes"
Sep 07 11:58:20.000299 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:20.000261    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:58:29.095444 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:29.095407    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:58:36.899348 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:36.899301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kx7sm"]
Sep 07 11:58:38.954397 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:38.954363    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6677b4689b-sx8j5_9eac05d8-8987-4b06-ada2-9d3c9de2c0b6/authorino/0.log"
Sep 07 11:58:45.703909 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:45.703824    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-g5gb8_1836dbd9-8f98-47ed-853a-2b660fa48a89/manager/0.log"
Sep 07 11:58:45.709603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:45.709582    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-g5gb8_1836dbd9-8f98-47ed-853a-2b660fa48a89/copy-manifests/0.log"
Sep 07 11:58:46.439603 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:46.439564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813025-wjrbg_9aafe403-36ae-4bbd-9f77-dc0c506e71cf/cleanup/0.log"
Sep 07 11:58:47.397013 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:47.396981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6677b4689b-sx8j5_9eac05d8-8987-4b06-ada2-9d3c9de2c0b6/authorino/0.log"
Sep 07 11:58:47.780519 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:47.780423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-7rk94_0d8261f2-86a8-44d8-afbf-575adedbf159/kuadrant-console-plugin/0.log"
Sep 07 11:58:48.043395 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:48.043303    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-kx7sm_4b145e05-ef9e-49ec-b44e-04d283e34fd3/limitador/0.log"
Sep 07 11:58:48.811262 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:48.811228    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-jxnn7_33e2bc5b-66b3-4090-865d-d4ea6e08e32c/kube-auth-proxy/0.log"
Sep 07 11:58:49.813582 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:49.813556    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zk2hl_d6c66737-a29b-46fb-9d79-6de216f7f94f/main/0.log"
Sep 07 11:58:51.135521 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:51.135456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:58:56.664040 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.664002    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9f2qg/must-gather-fm5k4"]
Sep 07 11:58:56.664568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.664552    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba2bf1d-61c8-4ba5-bd91-b78aff066405" containerName="maas-api"
Sep 07 11:58:56.664568 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.664570    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba2bf1d-61c8-4ba5-bd91-b78aff066405" containerName="maas-api"
Sep 07 11:58:56.664654 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.664645    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba2bf1d-61c8-4ba5-bd91-b78aff066405" containerName="maas-api"
Sep 07 11:58:56.667660 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.667643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:56.669978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.669943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f2qg\"/\"openshift-service-ca.crt\""
Sep 07 11:58:56.669978 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.669953    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9f2qg\"/\"default-dockercfg-svqxd\""
Sep 07 11:58:56.670800 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.670784    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f2qg\"/\"kube-root-ca.crt\""
Sep 07 11:58:56.676533 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.676511    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f2qg/must-gather-fm5k4"]
Sep 07 11:58:56.680054 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.680030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:58:56.741789 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.741752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gsh\" (UniqueName: \"kubernetes.io/projected/3ae78a45-3f1c-4e01-9369-a997d65d1742-kube-api-access-69gsh\") pod \"must-gather-fm5k4\" (UID: \"3ae78a45-3f1c-4e01-9369-a997d65d1742\") " pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:56.741789 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.741798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ae78a45-3f1c-4e01-9369-a997d65d1742-must-gather-output\") pod \"must-gather-fm5k4\" (UID: \"3ae78a45-3f1c-4e01-9369-a997d65d1742\") " pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:56.743297 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.743261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:58:56.842459 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.842428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ae78a45-3f1c-4e01-9369-a997d65d1742-must-gather-output\") pod \"must-gather-fm5k4\" (UID: \"3ae78a45-3f1c-4e01-9369-a997d65d1742\") " pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:56.842628 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.842556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gsh\" (UniqueName: \"kubernetes.io/projected/3ae78a45-3f1c-4e01-9369-a997d65d1742-kube-api-access-69gsh\") pod \"must-gather-fm5k4\" (UID: \"3ae78a45-3f1c-4e01-9369-a997d65d1742\") " pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:56.842780 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.842761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ae78a45-3f1c-4e01-9369-a997d65d1742-must-gather-output\") pod \"must-gather-fm5k4\" (UID: \"3ae78a45-3f1c-4e01-9369-a997d65d1742\") " pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:56.850765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.850733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gsh\" (UniqueName: \"kubernetes.io/projected/3ae78a45-3f1c-4e01-9369-a997d65d1742-kube-api-access-69gsh\") pod \"must-gather-fm5k4\" (UID: \"3ae78a45-3f1c-4e01-9369-a997d65d1742\") " pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:56.977216 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:56.977151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f2qg/must-gather-fm5k4"
Sep 07 11:58:57.247917 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:57.247890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f2qg/must-gather-fm5k4"]
Sep 07 11:58:57.249359 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:58:57.249331    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ae78a45_3f1c_4e01_9369_a997d65d1742.slice/crio-8a9c01b4926ae77ecd01238144656aeec6dedfc88bd8d277f78fe2b91263c2b6 WatchSource:0}: Error finding container 8a9c01b4926ae77ecd01238144656aeec6dedfc88bd8d277f78fe2b91263c2b6: Status 404 returned error can't find the container with id 8a9c01b4926ae77ecd01238144656aeec6dedfc88bd8d277f78fe2b91263c2b6
Sep 07 11:58:57.466077 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:57.466045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f2qg/must-gather-fm5k4" event={"ID":"3ae78a45-3f1c-4e01-9369-a997d65d1742","Type":"ContainerStarted","Data":"8a9c01b4926ae77ecd01238144656aeec6dedfc88bd8d277f78fe2b91263c2b6"}
Sep 07 11:58:58.475481 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:58.475426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f2qg/must-gather-fm5k4" event={"ID":"3ae78a45-3f1c-4e01-9369-a997d65d1742","Type":"ContainerStarted","Data":"69a47dc1c991ee8c7e34113e0fda72aad6113288f422cacd284cc56f74b93723"}
Sep 07 11:58:59.482650 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:59.482608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f2qg/must-gather-fm5k4" event={"ID":"3ae78a45-3f1c-4e01-9369-a997d65d1742","Type":"ContainerStarted","Data":"31d5719218bbaffa78c690a6ee377ebf53e23624552292819130d65e0bff65b3"}
Sep 07 11:58:59.499836 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:59.499786    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9f2qg/must-gather-fm5k4" podStartSLOduration=2.467283768 podStartE2EDuration="3.499771858s" totalImagesPullingTime="1.03248809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:58:59.499025632 +0000 UTC m=+2170.973873782" watchObservedRunningTime="2026-09-07 11:58:59.499771858 +0000 UTC m=+2170.974620007"
Sep 07 11:58:59.927108 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:59.927056    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sjsts_3a1df1aa-6e57-4a72-8f6f-06c8588fab09/global-pull-secret-syncer/0.log"
Sep 07 11:58:59.981378 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:58:59.981347    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log"
Sep 07 11:59:00.058294 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:00.058259    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-163.ec2.internal_07877c340319b68616068533f7d8229b/haproxy/0.log"
Sep 07 11:59:02.358688 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:02.358649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-47js8_32d447c3-1090-4e73-a7a4-c510811c26e3/konnectivity-agent/0.log" limit=1024
Sep 07 11:59:04.541541 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:04.541503    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6677b4689b-sx8j5_9eac05d8-8987-4b06-ada2-9d3c9de2c0b6/authorino/0.log"
Sep 07 11:59:04.614355 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:04.614315    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-7rk94_0d8261f2-86a8-44d8-afbf-575adedbf159/kuadrant-console-plugin/0.log"
Sep 07 11:59:04.681331 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:04.681305    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-kx7sm_4b145e05-ef9e-49ec-b44e-04d283e34fd3/limitador/0.log"
Sep 07 11:59:06.345317 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.345277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-v725n_fc1f1428-c3e4-4865-8bf8-e713111191a2/cluster-monitoring-operator/0.log"
Sep 07 11:59:06.371790 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.371752    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fvxtx_4b2024fd-cec9-4813-a943-d892c6382240/kube-state-metrics/0.log"
Sep 07 11:59:06.387536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.387505    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fvxtx_4b2024fd-cec9-4813-a943-d892c6382240/kube-rbac-proxy-main/0.log"
Sep 07 11:59:06.409313 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.409281    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fvxtx_4b2024fd-cec9-4813-a943-d892c6382240/kube-rbac-proxy-self/0.log"
Sep 07 11:59:06.434890 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.434857    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69df5b5b8d-ngz8w_bc3e7c79-39a0-4601-a6cd-6c6a5d424497/metrics-server/0.log"
Sep 07 11:59:06.631079 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.631045    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbvs9_00a2ba78-7842-49fb-b9a3-bf35a59f563b/node-exporter/0.log"
Sep 07 11:59:06.651254 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.651163    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbvs9_00a2ba78-7842-49fb-b9a3-bf35a59f563b/kube-rbac-proxy/0.log"
Sep 07 11:59:06.670318 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.670291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbvs9_00a2ba78-7842-49fb-b9a3-bf35a59f563b/init-textfile/0.log"
Sep 07 11:59:06.702080 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.702048    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fnmvd_2a4d9c60-09a2-450c-88c2-e5fe77f4a24d/kube-rbac-proxy-main/0.log"
Sep 07 11:59:06.720434 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.720395    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fnmvd_2a4d9c60-09a2-450c-88c2-e5fe77f4a24d/kube-rbac-proxy-self/0.log"
Sep 07 11:59:06.742032 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:06.742001    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fnmvd_2a4d9c60-09a2-450c-88c2-e5fe77f4a24d/openshift-state-metrics/0.log"
Sep 07 11:59:07.011298 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:07.011213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-sdsp8_1d5e3d1c-41be-47fc-9a87-d6af40e293e3/prometheus-operator/0.log"
Sep 07 11:59:07.041038 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:07.041004    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-sdsp8_1d5e3d1c-41be-47fc-9a87-d6af40e293e3/kube-rbac-proxy/0.log"
Sep 07 11:59:07.072893 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:07.072844    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-dkcws_233d9c7f-f265-4329-8870-f931167526ef/prometheus-operator-admission-webhook/0.log"
Sep 07 11:59:07.118421 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:07.118391    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79f946948-pvmxp_c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5/telemeter-client/0.log"
Sep 07 11:59:07.150415 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:07.150379    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79f946948-pvmxp_c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5/reload/0.log"
Sep 07 11:59:07.179854 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:07.179822    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79f946948-pvmxp_c2f2e815-c78e-4bc6-a8c8-63f3f7a7def5/kube-rbac-proxy/0.log"
Sep 07 11:59:09.372224 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.372191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/1.log"
Sep 07 11:59:09.382117 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.382088    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5xm2q_7954bc9a-b5a5-4829-aed7-d4004433d11c/console-operator/2.log"
Sep 07 11:59:09.845977 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.845884    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"]
Sep 07 11:59:09.851425 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.851396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:09.857201 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.857163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"]
Sep 07 11:59:09.982565 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.982522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-proc\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:09.982765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.982581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-lib-modules\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:09.982765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.982674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-sys\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:09.982765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.982735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkb49\" (UniqueName: \"kubernetes.io/projected/c6bf0f19-dd05-4a06-b996-84d788aeb523-kube-api-access-qkb49\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:09.982765 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.982760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-podres\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:09.999396 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:09.999351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-9xkx5_043202bc-f871-4192-9a8d-4bf4f0b8f662/download-server/0.log"
Sep 07 11:59:10.084159 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.084113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkb49\" (UniqueName: \"kubernetes.io/projected/c6bf0f19-dd05-4a06-b996-84d788aeb523-kube-api-access-qkb49\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.084361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.084177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-podres\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.084361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.084228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-proc\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.084361 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.084263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-lib-modules\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.084567 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.084387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-sys\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.084964 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.084939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-podres\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.085184 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.085161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-proc\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.085302 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.085205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-sys\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.085520 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.085504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6bf0f19-dd05-4a06-b996-84d788aeb523-lib-modules\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.095138 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.095108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkb49\" (UniqueName: \"kubernetes.io/projected/c6bf0f19-dd05-4a06-b996-84d788aeb523-kube-api-access-qkb49\") pod \"perf-node-gather-daemonset-dlfwd\" (UID: \"c6bf0f19-dd05-4a06-b996-84d788aeb523\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.168434 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.168399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:10.450755 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.450726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"]
Sep 07 11:59:10.452614 ip-10-0-129-163 kubenswrapper[2578]: W0907 11:59:10.452581    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6bf0f19_dd05_4a06_b996_84d788aeb523.slice/crio-da1c785896d1962cd4fef1f44a1da9c0ed3e9ccb132883075ef195e572c9943d WatchSource:0}: Error finding container da1c785896d1962cd4fef1f44a1da9c0ed3e9ccb132883075ef195e572c9943d: Status 404 returned error can't find the container with id da1c785896d1962cd4fef1f44a1da9c0ed3e9ccb132883075ef195e572c9943d
Sep 07 11:59:10.591601 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:10.591570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd" event={"ID":"c6bf0f19-dd05-4a06-b996-84d788aeb523","Type":"ContainerStarted","Data":"da1c785896d1962cd4fef1f44a1da9c0ed3e9ccb132883075ef195e572c9943d"}
Sep 07 11:59:11.601997 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:11.601961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd" event={"ID":"c6bf0f19-dd05-4a06-b996-84d788aeb523","Type":"ContainerStarted","Data":"ee36d726029115b8945db061a271381cff69d07aa356bc381d2f1ef784db2e1a"}
Sep 07 11:59:11.602442 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:11.602139    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:11.951880 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:11.951851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tv695_52727a38-dbef-437a-96ec-4401f1a6292e/dns/0.log"
Sep 07 11:59:11.968785 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:11.968755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tv695_52727a38-dbef-437a-96ec-4401f1a6292e/kube-rbac-proxy/0.log"
Sep 07 11:59:12.007175 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:12.007147    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hkpwd_50fe92fb-f492-45be-894d-7e4428d1985b/dns-node-resolver/0.log"
Sep 07 11:59:12.601440 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:12.601411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnkxf_ff53ccaf-e144-41af-8248-489446634ebd/node-ca/0.log"
Sep 07 11:59:13.873921 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:13.873886    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-jxnn7_33e2bc5b-66b3-4090-865d-d4ea6e08e32c/kube-auth-proxy/0.log"
Sep 07 11:59:14.835726 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:14.835688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2tfrr_21838826-4304-4ee2-a828-5e201e7ff2d9/serve-healthcheck-canary/0.log"
Sep 07 11:59:15.340526 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:15.340490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-z5hz6_4c3b1517-0672-4bff-9738-3bc5682ea071/insights-operator/0.log"
Sep 07 11:59:15.341327 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:15.341302    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-z5hz6_4c3b1517-0672-4bff-9738-3bc5682ea071/insights-operator/1.log"
Sep 07 11:59:15.360231 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:15.360197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qc9p_99eb80d4-4fa5-4248-ab8c-9e64574e4f7a/kube-rbac-proxy/0.log"
Sep 07 11:59:15.378674 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:15.378639    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qc9p_99eb80d4-4fa5-4248-ab8c-9e64574e4f7a/exporter/0.log"
Sep 07 11:59:15.399182 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:15.399155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qc9p_99eb80d4-4fa5-4248-ab8c-9e64574e4f7a/extractor/0.log"
Sep 07 11:59:17.620536 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:17.620489    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd"
Sep 07 11:59:17.641000 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:17.640945    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-dlfwd" podStartSLOduration=8.640924775 podStartE2EDuration="8.640924775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:59:11.616853991 +0000 UTC m=+2183.091702143" watchObservedRunningTime="2026-09-07 11:59:17.640924775 +0000 UTC m=+2189.115772924"
Sep 07 11:59:17.966863 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:17.966765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-g5gb8_1836dbd9-8f98-47ed-853a-2b660fa48a89/manager/0.log"
Sep 07 11:59:17.984853 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:17.984826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-g5gb8_1836dbd9-8f98-47ed-853a-2b660fa48a89/copy-manifests/0.log"
Sep 07 11:59:19.483120 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:19.483003    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79b5d8b54d-55ptn_bb334a9e-9a3d-4ab5-bc42-d39dc153b4d2/manager/0.log"
Sep 07 11:59:20.400844 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:20.400811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-jxnn7_33e2bc5b-66b3-4090-865d-d4ea6e08e32c/kube-auth-proxy/0.log"
Sep 07 11:59:22.331354 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:22.331318    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-jxnn7_33e2bc5b-66b3-4090-865d-d4ea6e08e32c/kube-auth-proxy/0.log"
Sep 07 11:59:24.945005 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:24.944971    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fpjjx_383ede9c-a994-4f54-88f9-c4647a3e6c8d/kube-storage-version-migrator-operator/1.log"
Sep 07 11:59:24.946822 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:24.946787    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fpjjx_383ede9c-a994-4f54-88f9-c4647a3e6c8d/kube-storage-version-migrator-operator/0.log"
Sep 07 11:59:26.034849 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.034823    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwz6m_4dba6b53-96bc-4321-8260-caa270ad6ca9/kube-multus-additional-cni-plugins/0.log"
Sep 07 11:59:26.051694 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.051656    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwz6m_4dba6b53-96bc-4321-8260-caa270ad6ca9/egress-router-binary-copy/0.log"
Sep 07 11:59:26.067805 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.067778    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwz6m_4dba6b53-96bc-4321-8260-caa270ad6ca9/cni-plugins/0.log"
Sep 07 11:59:26.085389 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.085365    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwz6m_4dba6b53-96bc-4321-8260-caa270ad6ca9/bond-cni-plugin/0.log"
Sep 07 11:59:26.101851 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.101825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwz6m_4dba6b53-96bc-4321-8260-caa270ad6ca9/routeoverride-cni/0.log"
Sep 07 11:59:26.118887 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.118862    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwz6m_4dba6b53-96bc-4321-8260-caa270ad6ca9/whereabouts-cni-bincopy/0.log"
Sep 07 11:59:26.135656 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.135634    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwz6m_4dba6b53-96bc-4321-8260-caa270ad6ca9/whereabouts-cni/0.log"
Sep 07 11:59:26.335003 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.334920    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k7j9h_006ca005-7640-42ec-86c4-7a55ca8ff076/kube-multus/0.log"
Sep 07 11:59:26.511535 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.511496    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lthxp_a90bddad-de76-4653-8a1b-e3546fb5aa7f/network-metrics-daemon/0.log"
Sep 07 11:59:26.525009 ip-10-0-129-163 kubenswrapper[2578]: I0907 11:59:26.524976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lthxp_a90bddad-de76-4653-8a1b-e3546fb5aa7f/kube-rbac-proxy/0.log"