Sep 10 23:35:57.913587 ip-10-0-128-88 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 23:35:58.373480 ip-10-0-128-88 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:35:58.373480 ip-10-0-128-88 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 23:35:58.373480 ip-10-0-128-88 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:35:58.373480 ip-10-0-128-88 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 23:35:58.373480 ip-10-0-128-88 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:35:58.374438 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.374355    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 23:35:58.379926 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379911    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379927    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379932    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379935    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379940    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379944    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379946    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379949    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379953    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379956    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379959    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379961    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379964    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379968    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:35:58.379964 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379971    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379974    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379976    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379985    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379987    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379990    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379993    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.379995    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380000    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380004    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380007    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380010    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380013    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380016    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380018    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380021    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380023    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380026    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380028    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:35:58.380319 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380031    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380034    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380036    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380039    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380041    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380044    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380046    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380050    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380052    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380055    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380057    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380060    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380062    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380065    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380067    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380070    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380073    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380076    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380078    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380081    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:35:58.380818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380083    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380086    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380088    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380091    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380094    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380096    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380099    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380101    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380104    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380107    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380110    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380112    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380115    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380118    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380121    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380124    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380126    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380129    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380131    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380133    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:35:58.381588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380136    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380139    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380141    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380144    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380146    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380149    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380151    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380155    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380158    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380160    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380163    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380166    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380169    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380171    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380174    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380176    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380180    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380184    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380186    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:35:58.382067 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380189    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:35:58.382541 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380192    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:35:58.382541 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380195    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:35:58.382541 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380197    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:35:58.382541 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380200    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:35:58.382541 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380202    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:35:58.382541 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.380204    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:35:58.383457 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383441    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:35:58.383457 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383457    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383461    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383465    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383469    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383472    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383476    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383478    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383481    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383484    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383487    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383491    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383495    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383498    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383501    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383504    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383506    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383509    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383512    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383514    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383517    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:35:58.383548 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383519    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383522    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383525    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383527    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383530    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383532    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383535    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383537    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383540    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383543    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383547    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383551    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383554    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383557    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383561    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383564    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383567    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383570    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383572    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:35:58.384042 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383575    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383577    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383580    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383582    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383585    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383587    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383590    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383593    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383596    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383598    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383601    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383603    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383606    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383608    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383611    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383614    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383617    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383619    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383622    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383625    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:35:58.384557 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383627    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383630    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383633    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383636    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383638    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383641    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383644    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383646    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383649    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383652    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383654    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383657    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383659    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383662    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383664    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383667    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383670    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383672    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383675    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383677    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:35:58.385044 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383680    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383682    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383685    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383688    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383691    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383693    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383696    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383699    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383703    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383706    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383708    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383711    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383713    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383716    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383719    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383721    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383724    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383726    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.383730    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:35:58.385547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384081    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384087    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384091    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384094    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384097    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384100    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384102    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384105    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384108    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384110    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384113    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384116    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384118    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384121    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384123    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384126    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384129    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384131    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384134    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384137    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:35:58.386012 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384140    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384143    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384145    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384148    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384150    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384153    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384155    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384158    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384160    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384163    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384165    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384168    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384170    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384173    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384175    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384178    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384180    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384182    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384185    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384187    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:35:58.386507 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384190    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384192    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384195    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384197    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384200    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384202    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384205    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384207    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384210    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384213    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384215    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384218    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384221    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384224    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384227    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384229    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384232    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384235    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384237    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384240    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:35:58.386978 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384243    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384245    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384248    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384251    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384254    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384256    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384259    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384261    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384264    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384266    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384269    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384285    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384288    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384290    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384293    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384297    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384300    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384304    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384307    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:35:58.387499 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384310    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384312    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384316    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384323    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384326    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384329    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384331    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384334    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384336    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384339    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384342    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384344    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384347    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384349    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384352    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384354    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384357    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384359    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.384362    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:35:58.387949 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384433    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384479    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384486    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384490    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384494    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384497    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384502    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384506    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384509    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384512    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384516    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384519    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384523    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384526    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384528    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384531    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384536    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384539    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384544    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384546    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384549    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384552    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384556    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384560    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 23:35:58.388445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384563    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384566    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384569    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384573    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384576    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384579    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384582    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384585    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384589    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384593    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384595    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384598    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384601    2578 flags.go:64] FLAG: --enable-server="true"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384604    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384608    2578 flags.go:64] FLAG: --event-burst="100"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384611    2578 flags.go:64] FLAG: --event-qps="50"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384614    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384617    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384620    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384624    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384627    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384630    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384633    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384636    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384638    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 23:35:58.389027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384642    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384645    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384648    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384650    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384653    2578 flags.go:64] FLAG: --feature-gates=""
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384657    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384660    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384664    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384667    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384670    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384673    2578 flags.go:64] FLAG: --help="false"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384676    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384679    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384682    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384685    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384689    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384692    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384695    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384697    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384700    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384703    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384706    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384709    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384712    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 10 23:35:58.389700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384715    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384718    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384721    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384724    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384727    2578 flags.go:64] FLAG: --lock-file=""
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384729    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384735    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384738    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384743    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384746    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384749    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384752    2578 flags.go:64] FLAG: --logging-format="text"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384755    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384758    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384761    2578 flags.go:64] FLAG: --manifest-url=""
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384764    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384768    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384771    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384775    2578 flags.go:64] FLAG: --max-pods="110"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384778    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384782    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384784    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384788    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384790    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384793    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 23:35:58.390290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384796    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384804    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384807    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384810    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384813    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384816    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384822    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384824    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384828    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384830    2578 flags.go:64] FLAG: --port="10250"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384833    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384836    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a8345345481fb5f"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384840    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384844    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384847    2578 flags.go:64] FLAG: --register-node="true"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384850    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384853    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384857    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384860    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384863    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384866    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384869    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384872    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384875    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384878    2578 flags.go:64] FLAG: --runonce="false"
Sep 10 23:35:58.390895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384881    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384883    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384886    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384889    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384892    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384895    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384898    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384901    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384904    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384906    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384910    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384914    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384917    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384920    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384923    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384928    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384930    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384933    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384938    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384941    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384945    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384948    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384951    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384954    2578 flags.go:64] FLAG: --v="2"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384957    2578 flags.go:64] FLAG: --version="false"
Sep 10 23:35:58.391522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384962    2578 flags.go:64] FLAG: --vmodule=""
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384966    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.384969    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385081    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385085    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385088    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385093    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385097    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385100    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385102    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385105    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385108    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385111    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385113    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385116    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385119    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385121    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385124    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385128    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:35:58.392142 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385131    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385134    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385137    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385140    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385142    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385145    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385148    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385151    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385154    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385157    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385160    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385163    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385165    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385168    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385171    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385174    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385176    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385179    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385182    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385184    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:35:58.392613 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385187    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385189    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385192    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385195    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385197    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385200    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385202    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385205    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385208    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385210    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385213    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385217    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385220    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385222    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385225    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385227    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385230    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385232    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385235    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:35:58.393097 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385237    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385242    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385244    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385247    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385249    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385252    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385254    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385257    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385266    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385269    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385284    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385287    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385290    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385293    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385297    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385301    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385304    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385307    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385309    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:35:58.393626 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385315    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385320    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385324    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385328    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385332    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385336    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385338    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385341    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385343    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385346    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385349    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385351    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385354    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385356    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385360    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385363    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385366    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385368    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385371    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385374    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:35:58.394089 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385376    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385379    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385383    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385385    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.385388    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.385393    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.392883    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.392897    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.392910    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.392917    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392971    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392975    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392979    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392982    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:35:58.394588 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392986    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392990    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392993    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392995    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.392998    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393001    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393004    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393008    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393010    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393013    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393015    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393019    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393023    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393037    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393040    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393043    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393046    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393049    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393051    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393054    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:35:58.394936 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393057    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393060    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393062    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393065    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393067    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393072    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393075    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393078    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393081    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393083    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393086    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393089    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393091    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393094    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393096    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393099    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393101    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393104    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393107    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:35:58.395519 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393110    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393113    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393117    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393120    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393123    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393127    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393130    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393132    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393135    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393138    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393142    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393145    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393147    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393150    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393152    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393155    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393158    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393160    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393163    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393166    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:35:58.395983 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393169    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393171    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393174    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393176    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393179    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393181    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393184    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393187    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393189    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393192    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393195    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393197    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393200    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393203    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393205    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393208    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393210    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393213    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393216    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393218    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:35:58.396497 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393221    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393224    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393227    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393229    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393232    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393234    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393237    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393240    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393242    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393245    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393247    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393250    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393253    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393255    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393258    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:35:58.396986 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393261    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.393265    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393387    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393392    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393395    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393398    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393401    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393404    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393406    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393409    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393412    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393414    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393417    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393419    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393422    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:35:58.397359 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393425    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393427    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393430    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393432    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393436    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393438    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393441    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393443    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393446    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393450    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393453    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393457    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393460    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393463    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393465    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393469    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393472    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393474    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393477    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:35:58.397745 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393479    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393482    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393485    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393487    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393490    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393492    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393495    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393498    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393500    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393503    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393505    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393515    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393517    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393520    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393523    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393525    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393528    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393530    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393533    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393536    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:35:58.398188 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393538    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393541    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393544    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393546    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393549    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393551    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393554    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393556    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393559    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393562    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393565    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393568    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393570    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393573    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393575    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393578    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393580    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393583    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393585    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393588    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:35:58.398691 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393590    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393593    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393595    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393599    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393602    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393605    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393608    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393611    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393614    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393617    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393619    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393622    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393625    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393628    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393631    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393633    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393636    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393638    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393641    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:35:58.399179 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393644    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393646    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393649    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393653    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393656    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393659    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393661    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:58.393664    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.393669    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.393764    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.396178    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.397292    2578 server.go:1019] "Starting client certificate rotation"
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.397387    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:35:58.399697 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.397430    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:35:58.400254 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.400240    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:35:58.401318 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.401300    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:35:58.408196 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.408177    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 10 23:35:58.411436 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.411420    2578 log.go:25] "Validated CRI v1 image API"
Sep 10 23:35:58.412194 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.412179    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 23:35:58.416107 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.416087    2578 fs.go:135] Filesystem UUIDs: map[6a074a06-1089-480f-ab54-f412d6ae0625:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f21320ae-0d17-4b75-a40a-513f5e541775:/dev/nvme0n1p3]
Sep 10 23:35:58.416182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.416105    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 23:35:58.422124 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.422012    2578 manager.go:216] Machine: {Timestamp:2026-09-10 23:35:58.41969204 +0000 UTC m=+0.375923834 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120881 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec228c3aa83be2313d0d50401f237223 SystemUUID:ec228c3a-a83b-e231-3d0d-50401f237223 BootID:cd8642fa-da7a-47be-88df-e9a2b56b384c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:aa:2e:20:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:aa:2e:20:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:65:8a:70:ef:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 23:35:58.422124 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.422116    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 23:35:58.422300 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.422217    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 23:35:58.424782 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.424755    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 23:35:58.424935 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.424785    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 23:35:58.426382 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.426369    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 23:35:58.426443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.426387    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 23:35:58.426443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.426398    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:35:58.427290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.427265    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:35:58.428527 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.428515    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 23:35:58.428723 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.428707    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:35:58.433037 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433023    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:35:58.433163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433151    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 23:35:58.433217 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433190    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 23:35:58.433217 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433210    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 23:35:58.433317 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433233    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 23:35:58.433317 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433247    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 10 23:35:58.433317 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433258    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 23:35:58.433714 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433695    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:35:58.433758 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.433723    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:35:58.436296 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.436268    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 23:35:58.439361 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.439347    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 23:35:58.439427 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.439376    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 23:35:58.441404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441387    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 23:35:58.441452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441419    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 23:35:58.441452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441433    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 23:35:58.441452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441444    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 23:35:58.441547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441462    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 23:35:58.441547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441475    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 23:35:58.441547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441494    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 23:35:58.441547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441503    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 23:35:58.441547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441522    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 23:35:58.441547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441530    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 23:35:58.441547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441543    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 23:35:58.441719 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.441551    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 23:35:58.442519 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.442506    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 23:35:58.442563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.442521    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 23:35:58.444691 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.444664    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 23:35:58.444759 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.444664    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 23:35:58.446505 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.446491    2578 server.go:1268] "Started kubelet"
Sep 10 23:35:58.446581 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.446527    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 23:35:58.452139 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.451800    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 23:35:58.452228 ip-10-0-128-88 systemd[1]: Started Kubernetes Kubelet.
Sep 10 23:35:58.453172 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.452315    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 23:35:58.455781 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.455540    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 23:35:58.457571 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.457556    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 23:35:58.458857 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.458832    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 23:35:58.459117 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.459094    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 23:35:58.460190 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.459108    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-88.ec2.internal.18d419d5e5c3a2c1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-88.ec2.internal,UID:ip-10-0-128-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-88.ec2.internal,},FirstTimestamp:2026-09-10 23:35:58.446461633 +0000 UTC m=+0.402693426,LastTimestamp:2026-09-10 23:35:58.446461633 +0000 UTC m=+0.402693426,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-88.ec2.internal,}"
Sep 10 23:35:58.463352 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.463333    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 23:35:58.463352 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.463344    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 23:35:58.463642 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.463525    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:58.464128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464077    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 23:35:58.464128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464093    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 23:35:58.464128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464103    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 23:35:58.464308 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464153    2578 factory.go:55] Registering systemd factory
Sep 10 23:35:58.464308 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464188    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 10 23:35:58.464308 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464211    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 23:35:58.464308 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464219    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 23:35:58.464484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464427    2578 factory.go:153] Registering CRI-O factory
Sep 10 23:35:58.464484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464440    2578 factory.go:223] Registration of the crio container factory successfully
Sep 10 23:35:58.464484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464485    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 23:35:58.464619 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464499    2578 factory.go:103] Registering Raw factory
Sep 10 23:35:58.464619 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464514    2578 manager.go:1196] Started watching for new ooms in manager
Sep 10 23:35:58.464877 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.464860    2578 manager.go:349] Starting recovery of all containers
Sep 10 23:35:58.465257 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.465231    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 23:35:58.465400 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.465356    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 23:35:58.470575 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.470553    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-57xt9"
Sep 10 23:35:58.477184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.477168    2578 manager.go:354] Recovery completed
Sep 10 23:35:58.478672 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.478544    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 23:35:58.482535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.482513    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-57xt9"
Sep 10 23:35:58.483377 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.483364    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:35:58.486001 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.485985    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:35:58.486088 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.486017    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:35:58.486088 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.486032    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:35:58.486684 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.486665    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 23:35:58.486684 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.486677    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:35:58.486814 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.486680    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 23:35:58.486814 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.486761    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:35:58.487452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.487434    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:35:58.487530 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.487463    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:35:58.487530 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.487476    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:35:58.488793 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.488778    2578 policy_none.go:49] "None policy: Start"
Sep 10 23:35:58.488793 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.488795    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 23:35:58.488889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.488805    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 23:35:58.490685 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.490673    2578 policy_none.go:47] "Start"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.516221    2578 manager.go:337] "Starting Device Plugin manager"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.516261    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.516291    2578 server.go:87] "Starting device plugin registration server"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.516620    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.516631    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.516765    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.516830    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.516838    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.517363    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.517406    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.526131    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.527299    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.527348    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.527372    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.527444    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 23:35:58.537641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.529226    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:35:58.617490 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.617463    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:35:58.618231 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.618217    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:35:58.618336 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.618250    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:35:58.618336 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.618265    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:35:58.618336 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.618307    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.626671 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.626657    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.626752 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.626682    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-88.ec2.internal\": node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:58.627522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.627498    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal"]
Sep 10 23:35:58.627573 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.627565    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:35:58.628558 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.628544    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:35:58.628642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.628572    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:35:58.628642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.628585    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:35:58.630879 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.630865    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:35:58.630982 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.630969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.630982 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.630995    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:35:58.631533 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.631504    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:35:58.631533 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.631526    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:35:58.631647 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.631538    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:35:58.631647 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.631550    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:35:58.631647 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.631553    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:35:58.631647 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.631561    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:35:58.633780 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.633767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.633829 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.633791    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:35:58.634398 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.634384    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:35:58.634446 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.634408    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:35:58.634446 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.634417    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:35:58.664451 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.664410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d5dcf442c60be7ea68574e057d3832-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal\" (UID: \"c3d5dcf442c60be7ea68574e057d3832\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.664451 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.664440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c87a324f1802bb8036fb2caa110222f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-88.ec2.internal\" (UID: \"0c87a324f1802bb8036fb2caa110222f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.664572 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.664464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3d5dcf442c60be7ea68574e057d3832-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal\" (UID: \"c3d5dcf442c60be7ea68574e057d3832\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.667807 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.667793    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-88.ec2.internal\" not found" node="ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.671902 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.671888    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-88.ec2.internal\" not found" node="ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.682318 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.682303    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:58.764701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.764640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d5dcf442c60be7ea68574e057d3832-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal\" (UID: \"c3d5dcf442c60be7ea68574e057d3832\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.764701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.764673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c87a324f1802bb8036fb2caa110222f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-88.ec2.internal\" (UID: \"0c87a324f1802bb8036fb2caa110222f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.764791 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.764708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3d5dcf442c60be7ea68574e057d3832-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal\" (UID: \"c3d5dcf442c60be7ea68574e057d3832\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.764791 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.764716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d5dcf442c60be7ea68574e057d3832-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal\" (UID: \"c3d5dcf442c60be7ea68574e057d3832\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.764791 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.764740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c87a324f1802bb8036fb2caa110222f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-88.ec2.internal\" (UID: \"0c87a324f1802bb8036fb2caa110222f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.764791 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.764744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3d5dcf442c60be7ea68574e057d3832-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal\" (UID: \"c3d5dcf442c60be7ea68574e057d3832\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.782784 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.782753    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:58.883180 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.883155    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:58.971804 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.971776    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.975576 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:58.975560    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:58.984086 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:58.984065    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:59.084719 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:59.084628    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:59.185208 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:59.185173    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:59.286563 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:59.286531    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:59.343472 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.343408    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:35:59.361535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.361509    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:35:59.387318 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:59.387286    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:59.397133 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.397111    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 23:35:59.397247 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.397230    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:35:59.397326 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.397274    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:35:59.397366 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.397299    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:35:59.488329 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:59.488298    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:59.488329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.488307    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 23:30:59 +0000 UTC" deadline="2028-03-04 15:27:45.772303888 +0000 UTC"
Sep 10 23:35:59.488527 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.488345    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12975h51m46.283966977s"
Sep 10 23:35:59.589870 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:35:59.589833    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-88.ec2.internal\" not found"
Sep 10 23:35:59.636559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.635663    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:35:59.664657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.664451    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:59.679463 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.679439    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:35:59.680541 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.680524    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal"
Sep 10 23:35:59.689815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.689788    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:35:59.790694 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:59.790660    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c87a324f1802bb8036fb2caa110222f.slice/crio-c186cc49608f7040abc32ecdf6fe4962e9701e32b4405ba171966bdd8a29ad84 WatchSource:0}: Error finding container c186cc49608f7040abc32ecdf6fe4962e9701e32b4405ba171966bdd8a29ad84: Status 404 returned error can't find the container with id c186cc49608f7040abc32ecdf6fe4962e9701e32b4405ba171966bdd8a29ad84
Sep 10 23:35:59.795065 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:35:59.795048    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:35:59.801510 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:35:59.801484    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d5dcf442c60be7ea68574e057d3832.slice/crio-23d78d141da02b93b15b0701eb901dbbaa2f1cba0dd2be7a68be07832f49980e WatchSource:0}: Error finding container 23d78d141da02b93b15b0701eb901dbbaa2f1cba0dd2be7a68be07832f49980e: Status 404 returned error can't find the container with id 23d78d141da02b93b15b0701eb901dbbaa2f1cba0dd2be7a68be07832f49980e
Sep 10 23:36:00.409391 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.409356    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:00.435919 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.435891    2578 apiserver.go:52] "Watching apiserver"
Sep 10 23:36:00.444116 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.444089    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 23:36:00.444820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.444795    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5lhn7","kube-system/konnectivity-agent-6bkhx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g","openshift-dns/node-resolver-bfnfj","openshift-image-registry/node-ca-zmbws","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal","openshift-multus/multus-9fm6t","openshift-multus/network-metrics-daemon-z6rwl","kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal","openshift-cluster-node-tuning-operator/tuned-spg86","openshift-multus/multus-additional-cni-plugins-qk2k7","openshift-network-diagnostics/network-check-target-vk9qm","openshift-network-operator/iptables-alerter-n7q2d","openshift-ovn-kubernetes/ovnkube-node-g8rhr"]
Sep 10 23:36:00.447745 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.447714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.451002 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.450600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.451387 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.451363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4vds2\""
Sep 10 23:36:00.452147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.451593    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 23:36:00.452147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.451711    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 23:36:00.452147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.451815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 23:36:00.452147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.452025    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 23:36:00.452914 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.452892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.453424 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.453401    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-szw92\""
Sep 10 23:36:00.453522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.453455    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:36:00.453522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.453489    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 23:36:00.453688 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.453641    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 23:36:00.456090 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.455310    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.456090 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.455459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 23:36:00.456090 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.455681    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 23:36:00.456090 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.455887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 23:36:00.456503 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.456114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vfpbn\""
Sep 10 23:36:00.460757 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.459405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.460757 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.460104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.461617    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.461810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x2bxs\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.462036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.462731    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.462922    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.463095    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmxbz\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.463254    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.463315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.463433    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.463684    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.463964    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.464125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.464261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.464388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.464508    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 23:36:00.464761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.464608    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wx56f\""
Sep 10 23:36:00.465608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.464782    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 23:36:00.465608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.465409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.465608 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.465499    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:00.468682 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.468024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.470859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.473232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7445n\""
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.473589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.473974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.474464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ssrcc\""
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.474802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-cni-multus\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.474840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d41773f0-a395-44dd-a28d-f8e3a3659411-host-slash\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.474869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae7c2aa-ebd2-4925-a132-c5b91088a016-tmp-dir\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.474901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-cni-netd\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.474936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqx9\" (UniqueName: \"kubernetes.io/projected/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-kube-api-access-qzqx9\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.474972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8x4t\" (UniqueName: \"kubernetes.io/projected/36b3a728-1f13-4576-963d-f0efea8f80d3-kube-api-access-g8x4t\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ae7c2aa-ebd2-4925-a132-c5b91088a016-hosts-file\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-dbus\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-kubelet\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-systemd\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-var-lib-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-slash\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.476797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/453364b7-bff4-4640-8fb2-07ba8b5f95c9-serviceca\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-cni-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-multus-certs\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kn5j\" (UniqueName: \"kubernetes.io/projected/d41773f0-a395-44dd-a28d-f8e3a3659411-kube-api-access-6kn5j\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-sys-fs\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-conf-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-etc-kubernetes\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d41773f0-a395-44dd-a28d-f8e3a3659411-iptables-alerter-script\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-registration-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-etc-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.475928    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7dba2314-a579-4782-8272-e1af9b28a637-ovn-node-metrics-cert\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-cni-bin\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-env-overrides\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.477948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-system-cni-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-daemon-config\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-ovn\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlrh2\" (UniqueName: \"kubernetes.io/projected/453364b7-bff4-4640-8fb2-07ba8b5f95c9-kube-api-access-jlrh2\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5236291c-b0c3-480c-bb27-f29ff23f8c42-cni-binary-copy\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-socket-dir-parent\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-k8s-cni-cncf-io\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-hostroot\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-log-socket\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/453364b7-bff4-4640-8fb2-07ba8b5f95c9-host\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.476994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-systemd-units\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-node-log\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-ovnkube-config\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-ovnkube-script-lib\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.478889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-socket-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-kubelet-config\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-cnibin\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-cni-bin\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.477928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-run-netns\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.478041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-device-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.478112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cplmx\" (UniqueName: \"kubernetes.io/projected/5236291c-b0c3-480c-bb27-f29ff23f8c42-kube-api-access-cplmx\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.478182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvqz\" (UniqueName: \"kubernetes.io/projected/8ae7c2aa-ebd2-4925-a132-c5b91088a016-kube-api-access-hgvqz\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.478247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-netns\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.478337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-kubelet\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.478399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.478442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5m8\" (UniqueName: \"kubernetes.io/projected/7dba2314-a579-4782-8272-e1af9b28a637-kube-api-access-zx5m8\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.479801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.479419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.481564 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.481540    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:36:00.481907 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.481884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t8hpb\""
Sep 10 23:36:00.482276 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.482255    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 23:36:00.483208 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.483162    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 23:36:00.484539 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.484510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:00.484634 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.484573    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:00.500600 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.500576    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jh8kw"
Sep 10 23:36:00.508664 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.508642    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jh8kw"
Sep 10 23:36:00.534166 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.534112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal" event={"ID":"c3d5dcf442c60be7ea68574e057d3832","Type":"ContainerStarted","Data":"23d78d141da02b93b15b0701eb901dbbaa2f1cba0dd2be7a68be07832f49980e"}
Sep 10 23:36:00.535730 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.535695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal" event={"ID":"0c87a324f1802bb8036fb2caa110222f","Type":"ContainerStarted","Data":"c186cc49608f7040abc32ecdf6fe4962e9701e32b4405ba171966bdd8a29ad84"}
Sep 10 23:36:00.565938 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.565917    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 23:36:00.579505 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.579629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ae7c2aa-ebd2-4925-a132-c5b91088a016-hosts-file\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.579629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-kubernetes\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.579629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-systemd\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.579629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-dbus\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.579629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-kubelet\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-systemd\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-var-lib-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-run\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-lib-modules\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-host\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-slash\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.579895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/453364b7-bff4-4640-8fb2-07ba8b5f95c9-serviceca\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-cni-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.579988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysconfig\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-system-cni-dir\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwjh\" (UniqueName: \"kubernetes.io/projected/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-kube-api-access-dhwjh\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-multus-certs\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn5j\" (UniqueName: \"kubernetes.io/projected/d41773f0-a395-44dd-a28d-f8e3a3659411-kube-api-access-6kn5j\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-sys-fs\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-conf-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-etc-kubernetes\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d41773f0-a395-44dd-a28d-f8e3a3659411-iptables-alerter-script\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-registration-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-modprobe-d\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-etc-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.580386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a18d1704-9495-49e1-912d-b1da73e3da26-agent-certs\") pod \"konnectivity-agent-6bkhx\" (UID: \"a18d1704-9495-49e1-912d-b1da73e3da26\") " pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-var-lib-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-systemd\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-conf-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-cni-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-etc-kubernetes\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ae7c2aa-ebd2-4925-a132-c5b91088a016-hosts-file\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-sys-fs\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-registration-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-slash\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-etc-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-dbus\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-kubelet\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.580848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-multus-certs\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d41773f0-a395-44dd-a28d-f8e3a3659411-iptables-alerter-script\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.581225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/453364b7-bff4-4640-8fb2-07ba8b5f95c9-serviceca\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cnibin\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7dba2314-a579-4782-8272-e1af9b28a637-ovn-node-metrics-cert\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysctl-d\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-var-lib-kubelet\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-cni-bin\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-env-overrides\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-system-cni-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-daemon-config\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-ovn\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrh2\" (UniqueName: \"kubernetes.io/projected/453364b7-bff4-4640-8fb2-07ba8b5f95c9-kube-api-access-jlrh2\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5236291c-b0c3-480c-bb27-f29ff23f8c42-cni-binary-copy\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-socket-dir-parent\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.582192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-k8s-cni-cncf-io\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-hostroot\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-log-socket\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/453364b7-bff4-4640-8fb2-07ba8b5f95c9-host\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-systemd-units\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-node-log\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-ovnkube-config\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-ovnkube-script-lib\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-socket-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-kubelet-config\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-cnibin\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-cni-bin\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-run-netns\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.581994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-device-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cplmx\" (UniqueName: \"kubernetes.io/projected/5236291c-b0c3-480c-bb27-f29ff23f8c42-kube-api-access-cplmx\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvqz\" (UniqueName: \"kubernetes.io/projected/8ae7c2aa-ebd2-4925-a132-c5b91088a016-kube-api-access-hgvqz\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.583054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-netns\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-kubelet\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5m8\" (UniqueName: \"kubernetes.io/projected/7dba2314-a579-4782-8272-e1af9b28a637-kube-api-access-zx5m8\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1704-9495-49e1-912d-b1da73e3da26-konnectivity-ca\") pod \"konnectivity-agent-6bkhx\" (UID: \"a18d1704-9495-49e1-912d-b1da73e3da26\") " pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-sys\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysctl-conf\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d667255-226d-49be-a363-8cb6b3bc750d-etc-tuned\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/453364b7-bff4-4640-8fb2-07ba8b5f95c9-host\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-system-cni-dir\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d667255-226d-49be-a363-8cb6b3bc750d-tmp\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qr2z\" (UniqueName: \"kubernetes.io/projected/3d667255-226d-49be-a363-8cb6b3bc750d-kube-api-access-8qr2z\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-cni-multus\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d41773f0-a395-44dd-a28d-f8e3a3659411-host-slash\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-run-netns\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-cni-multus\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-systemd-units\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.583944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.582870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-node-log\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-socket-dir-parent\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-hostroot\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-log-socket\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-netns\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-run-k8s-cni-cncf-io\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-kubelet-config\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-ovn\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-kubelet\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d41773f0-a395-44dd-a28d-f8e3a3659411-host-slash\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-device-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5236291c-b0c3-480c-bb27-f29ff23f8c42-cni-binary-copy\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-run-openvswitch\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36b3a728-1f13-4576-963d-f0efea8f80d3-socket-dir\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-host-var-lib-cni-bin\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae7c2aa-ebd2-4925-a132-c5b91088a016-tmp-dir\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5236291c-b0c3-480c-bb27-f29ff23f8c42-cnibin\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-cni-netd\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.584640 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.583506    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-ovnkube-config\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-cni-bin\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.583525    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.583596    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:01.083568979 +0000 UTC m=+3.039800782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.583756    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret podName:5b828bf5-24ab-4b4c-8d09-2279c1c4c709 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:01.083728549 +0000 UTC m=+3.039960556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret") pod "global-pull-secret-syncer-5lhn7" (UID: "5b828bf5-24ab-4b4c-8d09-2279c1c4c709") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7dba2314-a579-4782-8272-e1af9b28a637-host-cni-netd\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqx9\" (UniqueName: \"kubernetes.io/projected/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-kube-api-access-qzqx9\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.583926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8x4t\" (UniqueName: \"kubernetes.io/projected/36b3a728-1f13-4576-963d-f0efea8f80d3-kube-api-access-g8x4t\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.584102    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.584254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-ovnkube-script-lib\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.584608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae7c2aa-ebd2-4925-a132-c5b91088a016-tmp-dir\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.584967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7dba2314-a579-4782-8272-e1af9b28a637-env-overrides\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.585455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.585447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5236291c-b0c3-480c-bb27-f29ff23f8c42-multus-daemon-config\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.589577 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.589521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7dba2314-a579-4782-8272-e1af9b28a637-ovn-node-metrics-cert\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.591657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.591633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kn5j\" (UniqueName: \"kubernetes.io/projected/d41773f0-a395-44dd-a28d-f8e3a3659411-kube-api-access-6kn5j\") pod \"iptables-alerter-n7q2d\" (UID: \"d41773f0-a395-44dd-a28d-f8e3a3659411\") " pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.611466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.611440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5m8\" (UniqueName: \"kubernetes.io/projected/7dba2314-a579-4782-8272-e1af9b28a637-kube-api-access-zx5m8\") pod \"ovnkube-node-g8rhr\" (UID: \"7dba2314-a579-4782-8272-e1af9b28a637\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.612389 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.612342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqx9\" (UniqueName: \"kubernetes.io/projected/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-kube-api-access-qzqx9\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:00.612489 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.612407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvqz\" (UniqueName: \"kubernetes.io/projected/8ae7c2aa-ebd2-4925-a132-c5b91088a016-kube-api-access-hgvqz\") pod \"node-resolver-bfnfj\" (UID: \"8ae7c2aa-ebd2-4925-a132-c5b91088a016\") " pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.613471 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.613445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrh2\" (UniqueName: \"kubernetes.io/projected/453364b7-bff4-4640-8fb2-07ba8b5f95c9-kube-api-access-jlrh2\") pod \"node-ca-zmbws\" (UID: \"453364b7-bff4-4640-8fb2-07ba8b5f95c9\") " pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.614295 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.614269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cplmx\" (UniqueName: \"kubernetes.io/projected/5236291c-b0c3-480c-bb27-f29ff23f8c42-kube-api-access-cplmx\") pod \"multus-9fm6t\" (UID: \"5236291c-b0c3-480c-bb27-f29ff23f8c42\") " pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.615333 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.615310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8x4t\" (UniqueName: \"kubernetes.io/projected/36b3a728-1f13-4576-963d-f0efea8f80d3-kube-api-access-g8x4t\") pod \"aws-ebs-csi-driver-node-7p24g\" (UID: \"36b3a728-1f13-4576-963d-f0efea8f80d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.685440 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-kubernetes\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685440 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-systemd\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685440 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-run\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-lib-modules\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-host\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-kubernetes\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-systemd\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-run\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysconfig\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-system-cni-dir\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysconfig\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.685695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwjh\" (UniqueName: \"kubernetes.io/projected/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-kube-api-access-dhwjh\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-modprobe-d\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a18d1704-9495-49e1-912d-b1da73e3da26-agent-certs\") pod \"konnectivity-agent-6bkhx\" (UID: \"a18d1704-9495-49e1-912d-b1da73e3da26\") " pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cnibin\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysctl-d\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-var-lib-kubelet\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-lib-modules\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.685989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1704-9495-49e1-912d-b1da73e3da26-konnectivity-ca\") pod \"konnectivity-agent-6bkhx\" (UID: \"a18d1704-9495-49e1-912d-b1da73e3da26\") " pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-sys\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-host\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysctl-conf\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d667255-226d-49be-a363-8cb6b3bc750d-etc-tuned\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d667255-226d-49be-a363-8cb6b3bc750d-tmp\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.686222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qr2z\" (UniqueName: \"kubernetes.io/projected/3d667255-226d-49be-a363-8cb6b3bc750d-kube-api-access-8qr2z\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.687061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.687061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-modprobe-d\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.687061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.687061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.686870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-system-cni-dir\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.687061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.687034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysctl-d\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.687329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.687174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-etc-sysctl-conf\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.687329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.687037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cnibin\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.687463 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.687339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.687520 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.687463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-var-lib-kubelet\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.687882 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.687568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1704-9495-49e1-912d-b1da73e3da26-konnectivity-ca\") pod \"konnectivity-agent-6bkhx\" (UID: \"a18d1704-9495-49e1-912d-b1da73e3da26\") " pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.687882 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.687609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d667255-226d-49be-a363-8cb6b3bc750d-sys\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.688075 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.688052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.689642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.689580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a18d1704-9495-49e1-912d-b1da73e3da26-agent-certs\") pod \"konnectivity-agent-6bkhx\" (UID: \"a18d1704-9495-49e1-912d-b1da73e3da26\") " pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.689642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.689586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d667255-226d-49be-a363-8cb6b3bc750d-etc-tuned\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.690195 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.690172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d667255-226d-49be-a363-8cb6b3bc750d-tmp\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.697712 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.696447    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:00.697712 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.696474    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:00.697712 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.696488    2578 projected.go:196] Error preparing data for projected volume kube-api-access-k7br6 for pod openshift-network-diagnostics/network-check-target-vk9qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:00.697712 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:00.696561    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6 podName:7155a075-78c6-484c-9324-2f375d1ad2ac nodeName:}" failed. No retries permitted until 2026-09-10 23:36:01.196542025 +0000 UTC m=+3.152773831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k7br6" (UniqueName: "kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6") pod "network-check-target-vk9qm" (UID: "7155a075-78c6-484c-9324-2f375d1ad2ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:00.699361 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.699330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qr2z\" (UniqueName: \"kubernetes.io/projected/3d667255-226d-49be-a363-8cb6b3bc750d-kube-api-access-8qr2z\") pod \"tuned-spg86\" (UID: \"3d667255-226d-49be-a363-8cb6b3bc750d\") " pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.700791 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.700766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwjh\" (UniqueName: \"kubernetes.io/projected/4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d-kube-api-access-dhwjh\") pod \"multus-additional-cni-plugins-qk2k7\" (UID: \"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d\") " pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:00.768523 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.768496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fm6t"
Sep 10 23:36:00.782904 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.782882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n7q2d"
Sep 10 23:36:00.783704 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:00.783674    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5236291c_b0c3_480c_bb27_f29ff23f8c42.slice/crio-f7ecc137f7d78f308f6d7a4b94e58c8d749cceeb19aa312400ee10055ec24467 WatchSource:0}: Error finding container f7ecc137f7d78f308f6d7a4b94e58c8d749cceeb19aa312400ee10055ec24467: Status 404 returned error can't find the container with id f7ecc137f7d78f308f6d7a4b94e58c8d749cceeb19aa312400ee10055ec24467
Sep 10 23:36:00.792487 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:00.792449    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41773f0_a395_44dd_a28d_f8e3a3659411.slice/crio-b225da7823e5805c9be26e25cd73b4407c315aedc4ac4326963ad3230343c851 WatchSource:0}: Error finding container b225da7823e5805c9be26e25cd73b4407c315aedc4ac4326963ad3230343c851: Status 404 returned error can't find the container with id b225da7823e5805c9be26e25cd73b4407c315aedc4ac4326963ad3230343c851
Sep 10 23:36:00.794934 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.794533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g"
Sep 10 23:36:00.802913 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.802885    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bfnfj"
Sep 10 23:36:00.804711 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:00.804685    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b3a728_1f13_4576_963d_f0efea8f80d3.slice/crio-67cbc615ea21742221d7f35f4c664b41d0b1cdac7c02fc33abb39c814e4a3052 WatchSource:0}: Error finding container 67cbc615ea21742221d7f35f4c664b41d0b1cdac7c02fc33abb39c814e4a3052: Status 404 returned error can't find the container with id 67cbc615ea21742221d7f35f4c664b41d0b1cdac7c02fc33abb39c814e4a3052
Sep 10 23:36:00.810420 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.810399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmbws"
Sep 10 23:36:00.812903 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:00.812878    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae7c2aa_ebd2_4925_a132_c5b91088a016.slice/crio-5a3b23c6d9d3cf9cee80ac4ebba33b30e6282a7c27e6050572e9dc2a2af30ea6 WatchSource:0}: Error finding container 5a3b23c6d9d3cf9cee80ac4ebba33b30e6282a7c27e6050572e9dc2a2af30ea6: Status 404 returned error can't find the container with id 5a3b23c6d9d3cf9cee80ac4ebba33b30e6282a7c27e6050572e9dc2a2af30ea6
Sep 10 23:36:00.818468 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.818446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:00.819958 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:00.819934    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453364b7_bff4_4640_8fb2_07ba8b5f95c9.slice/crio-d2cc44288184bfe67b4d5ce19018081376de0ff7d4617c4793afc5c885b321e9 WatchSource:0}: Error finding container d2cc44288184bfe67b4d5ce19018081376de0ff7d4617c4793afc5c885b321e9: Status 404 returned error can't find the container with id d2cc44288184bfe67b4d5ce19018081376de0ff7d4617c4793afc5c885b321e9
Sep 10 23:36:00.826589 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.826566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:00.836585 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.836564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-spg86"
Sep 10 23:36:00.844652 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:00.844630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qk2k7"
Sep 10 23:36:01.089264 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.089160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:01.089264 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.089218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:01.089516 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.089424    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:01.089576 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.089491    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret podName:5b828bf5-24ab-4b4c-8d09-2279c1c4c709 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:02.089471604 +0000 UTC m=+4.045703391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret") pod "global-pull-secret-syncer-5lhn7" (UID: "5b828bf5-24ab-4b4c-8d09-2279c1c4c709") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:01.089849 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.089819    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:01.089982 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.089904    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:02.089876276 +0000 UTC m=+4.046108071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:01.290550 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.290507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:01.290720 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.290663    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:01.290720 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.290689    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:01.290720 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.290702    2578 projected.go:196] Error preparing data for projected volume kube-api-access-k7br6 for pod openshift-network-diagnostics/network-check-target-vk9qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:01.290900 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:01.290773    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6 podName:7155a075-78c6-484c-9324-2f375d1ad2ac nodeName:}" failed. No retries permitted until 2026-09-10 23:36:02.290751985 +0000 UTC m=+4.246983792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7br6" (UniqueName: "kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6") pod "network-check-target-vk9qm" (UID: "7155a075-78c6-484c-9324-2f375d1ad2ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:01.512413 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.512257    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:01 +0000 UTC" deadline="2028-06-23 18:43:35.66586475 +0000 UTC"
Sep 10 23:36:01.512413 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.512312    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15643h7m34.15355803s"
Sep 10 23:36:01.539148 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.539101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmbws" event={"ID":"453364b7-bff4-4640-8fb2-07ba8b5f95c9","Type":"ContainerStarted","Data":"d2cc44288184bfe67b4d5ce19018081376de0ff7d4617c4793afc5c885b321e9"}
Sep 10 23:36:01.541278 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.541239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bfnfj" event={"ID":"8ae7c2aa-ebd2-4925-a132-c5b91088a016","Type":"ContainerStarted","Data":"5a3b23c6d9d3cf9cee80ac4ebba33b30e6282a7c27e6050572e9dc2a2af30ea6"}
Sep 10 23:36:01.542990 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.542958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g" event={"ID":"36b3a728-1f13-4576-963d-f0efea8f80d3","Type":"ContainerStarted","Data":"67cbc615ea21742221d7f35f4c664b41d0b1cdac7c02fc33abb39c814e4a3052"}
Sep 10 23:36:01.544674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.544647    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n7q2d" event={"ID":"d41773f0-a395-44dd-a28d-f8e3a3659411","Type":"ContainerStarted","Data":"b225da7823e5805c9be26e25cd73b4407c315aedc4ac4326963ad3230343c851"}
Sep 10 23:36:01.546405 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.546378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fm6t" event={"ID":"5236291c-b0c3-480c-bb27-f29ff23f8c42","Type":"ContainerStarted","Data":"f7ecc137f7d78f308f6d7a4b94e58c8d749cceeb19aa312400ee10055ec24467"}
Sep 10 23:36:01.843220 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:01.842919    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:02.098463 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.098376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:02.098463 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.098430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:02.098707 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.098567    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:02.098707 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.098631    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret podName:5b828bf5-24ab-4b4c-8d09-2279c1c4c709 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:04.098613518 +0000 UTC m=+6.054845304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret") pod "global-pull-secret-syncer-5lhn7" (UID: "5b828bf5-24ab-4b4c-8d09-2279c1c4c709") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:02.098707 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.098565    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:02.098882 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.098726    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:04.098704507 +0000 UTC m=+6.054936297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:02.194800 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.194756    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:02.197717 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:02.197470    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dba2314_a579_4782_8272_e1af9b28a637.slice/crio-1c296f922ee6a6b213f31cbb6fb6f72e7d689b2ab5c4e958d474760c9c4844ef WatchSource:0}: Error finding container 1c296f922ee6a6b213f31cbb6fb6f72e7d689b2ab5c4e958d474760c9c4844ef: Status 404 returned error can't find the container with id 1c296f922ee6a6b213f31cbb6fb6f72e7d689b2ab5c4e958d474760c9c4844ef
Sep 10 23:36:02.206074 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:02.206041    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d667255_226d_49be_a363_8cb6b3bc750d.slice/crio-42217ea4a47b44a807e80ec17e9bfc1054de8590e1dd6d384bc490a911c48637 WatchSource:0}: Error finding container 42217ea4a47b44a807e80ec17e9bfc1054de8590e1dd6d384bc490a911c48637: Status 404 returned error can't find the container with id 42217ea4a47b44a807e80ec17e9bfc1054de8590e1dd6d384bc490a911c48637
Sep 10 23:36:02.209900 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:02.209870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdd04d5_18fa_41e3_a658_1fe35ee0ad8d.slice/crio-f7bfb9dad5ca54ee9f1fe1911afc5b6a9eda83758471477509d228faeca84601 WatchSource:0}: Error finding container f7bfb9dad5ca54ee9f1fe1911afc5b6a9eda83758471477509d228faeca84601: Status 404 returned error can't find the container with id f7bfb9dad5ca54ee9f1fe1911afc5b6a9eda83758471477509d228faeca84601
Sep 10 23:36:02.299903 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.299617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:02.299903 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.299788    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:02.299903 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.299811    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:02.299903 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.299825    2578 projected.go:196] Error preparing data for projected volume kube-api-access-k7br6 for pod openshift-network-diagnostics/network-check-target-vk9qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:02.299903 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.299878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6 podName:7155a075-78c6-484c-9324-2f375d1ad2ac nodeName:}" failed. No retries permitted until 2026-09-10 23:36:04.299860224 +0000 UTC m=+6.256092022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7br6" (UniqueName: "kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6") pod "network-check-target-vk9qm" (UID: "7155a075-78c6-484c-9324-2f375d1ad2ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:02.512824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.512705    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:01 +0000 UTC" deadline="2028-02-11 14:55:53.100653344 +0000 UTC"
Sep 10 23:36:02.512824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.512773    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12447h19m50.587885581s"
Sep 10 23:36:02.528726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.528693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:02.528867 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.528843    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:02.529319 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.529297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:02.529455 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.529432    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:02.529631 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.529613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:02.529880 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:02.529707    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:02.551725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.551687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal" event={"ID":"0c87a324f1802bb8036fb2caa110222f","Type":"ContainerStarted","Data":"f607c70d4019b319627243a3ddb91ca61700d9a4adb6f84dd004383ad3d3d57e"}
Sep 10 23:36:02.556756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.556217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerStarted","Data":"f7bfb9dad5ca54ee9f1fe1911afc5b6a9eda83758471477509d228faeca84601"}
Sep 10 23:36:02.558100 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.558067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bkhx" event={"ID":"a18d1704-9495-49e1-912d-b1da73e3da26","Type":"ContainerStarted","Data":"7d535a2213bde1c97a50450ad0a15e863c4d110d62e6cc9c201ad3dd8e5201d7"}
Sep 10 23:36:02.561706 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.561641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-spg86" event={"ID":"3d667255-226d-49be-a363-8cb6b3bc750d","Type":"ContainerStarted","Data":"42217ea4a47b44a807e80ec17e9bfc1054de8590e1dd6d384bc490a911c48637"}
Sep 10 23:36:02.564666 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:02.564615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"1c296f922ee6a6b213f31cbb6fb6f72e7d689b2ab5c4e958d474760c9c4844ef"}
Sep 10 23:36:03.569743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:03.569291    2578 generic.go:356] "Generic (PLEG): container finished" podID="c3d5dcf442c60be7ea68574e057d3832" containerID="3dbfac0c9625a1b8bc7b3452fb6596077042da134f2193b52751347b42519e24" exitCode=0
Sep 10 23:36:03.569743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:03.569503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal" event={"ID":"c3d5dcf442c60be7ea68574e057d3832","Type":"ContainerDied","Data":"3dbfac0c9625a1b8bc7b3452fb6596077042da134f2193b52751347b42519e24"}
Sep 10 23:36:03.589066 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:03.589011    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-88.ec2.internal" podStartSLOduration=3.588990768 podStartE2EDuration="3.588990768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:02.567783906 +0000 UTC m=+4.524015714" watchObservedRunningTime="2026-09-10 23:36:03.588990768 +0000 UTC m=+5.545222575"
Sep 10 23:36:04.115530 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:04.115491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:04.115731 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:04.115548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:04.115794 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.115731    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:04.115862 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.115795    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret podName:5b828bf5-24ab-4b4c-8d09-2279c1c4c709 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:08.115774077 +0000 UTC m=+10.072005876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret") pod "global-pull-secret-syncer-5lhn7" (UID: "5b828bf5-24ab-4b4c-8d09-2279c1c4c709") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:04.115862 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.115829    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:04.115979 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.115869    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:08.115857611 +0000 UTC m=+10.072089400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:04.317720 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:04.317678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:04.317934 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.317917    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:04.318009 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.317942    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:04.318009 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.317955    2578 projected.go:196] Error preparing data for projected volume kube-api-access-k7br6 for pod openshift-network-diagnostics/network-check-target-vk9qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:04.318121 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.318015    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6 podName:7155a075-78c6-484c-9324-2f375d1ad2ac nodeName:}" failed. No retries permitted until 2026-09-10 23:36:08.317996315 +0000 UTC m=+10.274228108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7br6" (UniqueName: "kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6") pod "network-check-target-vk9qm" (UID: "7155a075-78c6-484c-9324-2f375d1ad2ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:04.531036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:04.530960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:04.531214 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.531114    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:04.531565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:04.531544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:04.531687 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.531664    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:04.531748 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:04.531734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:04.531824 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:04.531806    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:06.530437 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:06.530340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:06.530896 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:06.530494    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:06.530896 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:06.530879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:06.531012 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:06.530986    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:06.531067 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:06.531032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:06.531124 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:06.531106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:08.150068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:08.150016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:08.150068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:08.150067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:08.150997 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.150235    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:08.150997 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.150307    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret podName:5b828bf5-24ab-4b4c-8d09-2279c1c4c709 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:16.150286698 +0000 UTC m=+18.106518504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret") pod "global-pull-secret-syncer-5lhn7" (UID: "5b828bf5-24ab-4b4c-8d09-2279c1c4c709") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:08.150997 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.150409    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:08.150997 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.150480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:16.150461685 +0000 UTC m=+18.106693472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:08.352574 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:08.352529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:08.352768 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.352705    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:08.352768 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.352731    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:08.352768 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.352744    2578 projected.go:196] Error preparing data for projected volume kube-api-access-k7br6 for pod openshift-network-diagnostics/network-check-target-vk9qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:08.352938 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.352813    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6 podName:7155a075-78c6-484c-9324-2f375d1ad2ac nodeName:}" failed. No retries permitted until 2026-09-10 23:36:16.352791601 +0000 UTC m=+18.309023386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7br6" (UniqueName: "kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6") pod "network-check-target-vk9qm" (UID: "7155a075-78c6-484c-9324-2f375d1ad2ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:08.528674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:08.528647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:08.529132 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.529105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:08.529725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:08.529699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:08.529842 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.529818    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:08.530001 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:08.529976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:08.530116 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:08.530085    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:10.529123 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:10.528594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:10.529123 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:10.528627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:10.529123 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:10.528744    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:10.529123 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:10.528806    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:10.529123 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:10.528854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:10.529123 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:10.528906    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:12.527978 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:12.527934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:12.528437 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:12.527934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:12.528437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:12.528081    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:12.528437 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:12.528102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:12.528437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:12.528210    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:12.528437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:12.528317    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:14.527798 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:14.527770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:14.528222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:14.527773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:14.528222 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:14.527875    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:14.528222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:14.527774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:14.528222 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:14.527975    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:14.528222 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:14.528075    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:16.211858 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:16.211822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:16.212394 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:16.211868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:16.212394 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.211984    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:16.212394 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.212010    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:16.212394 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.212067    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:32.212044751 +0000 UTC m=+34.168276532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:16.212394 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.212089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret podName:5b828bf5-24ab-4b4c-8d09-2279c1c4c709 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:32.212079012 +0000 UTC m=+34.168310795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret") pod "global-pull-secret-syncer-5lhn7" (UID: "5b828bf5-24ab-4b4c-8d09-2279c1c4c709") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:16.413357 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:16.413316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:16.413533 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.413512    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:16.413573 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.413547    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:16.413573 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.413560    2578 projected.go:196] Error preparing data for projected volume kube-api-access-k7br6 for pod openshift-network-diagnostics/network-check-target-vk9qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:16.413635 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.413626    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6 podName:7155a075-78c6-484c-9324-2f375d1ad2ac nodeName:}" failed. No retries permitted until 2026-09-10 23:36:32.413606968 +0000 UTC m=+34.369838753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7br6" (UniqueName: "kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6") pod "network-check-target-vk9qm" (UID: "7155a075-78c6-484c-9324-2f375d1ad2ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:16.528042 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:16.528012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:16.528201 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:16.528012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:16.528201 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:16.528141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:16.528201 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.528138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:16.528365 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.528256    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:16.528425 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:16.528368    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:18.530778 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:18.530749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:18.531232 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:18.530861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:18.531232 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:18.530886    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:18.531232 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:18.530938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:18.531232 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:18.530956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:18.531232 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:18.531047    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:19.605608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.605253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"458d28a70a1beae7a9ade8b93f0604dba3b0faebf04e8088c03371d46d275058"}
Sep 10 23:36:19.606727 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.606702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmbws" event={"ID":"453364b7-bff4-4640-8fb2-07ba8b5f95c9","Type":"ContainerStarted","Data":"e243367e112d7fd2b4c84d38af6cbb06075c91ebcb24b6e6eb0c575719ab2bb6"}
Sep 10 23:36:19.608118 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.608099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g" event={"ID":"36b3a728-1f13-4576-963d-f0efea8f80d3","Type":"ContainerStarted","Data":"d2b29962d1bf6bda1f346dfd2af0fafc67ac835d5adcae9fbe008a112e9a8cd0"}
Sep 10 23:36:19.609363 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.609343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fm6t" event={"ID":"5236291c-b0c3-480c-bb27-f29ff23f8c42","Type":"ContainerStarted","Data":"db0fc160ca82be15a3c984204cf5cbff7390f670f0fe847f68dfa4d3d942a99c"}
Sep 10 23:36:19.611288 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.611259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal" event={"ID":"c3d5dcf442c60be7ea68574e057d3832","Type":"ContainerStarted","Data":"7887fef46c75138c3ea1311ddc6fd1c6077595d8eab5600fa5082fdfaa76491d"}
Sep 10 23:36:19.612608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.612588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bfnfj" event={"ID":"8ae7c2aa-ebd2-4925-a132-c5b91088a016","Type":"ContainerStarted","Data":"7ba77e96a67fde4fa642e2fb53305713450ccbca08b6ff5fa7e3ca4e5d3e0ab5"}
Sep 10 23:36:19.613983 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.613964    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d" containerID="48700563728b34d7c19e2eb4f0b19dc3effaa3385ba17947b86b1e488a2ef69c" exitCode=0
Sep 10 23:36:19.614049 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.614021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerDied","Data":"48700563728b34d7c19e2eb4f0b19dc3effaa3385ba17947b86b1e488a2ef69c"}
Sep 10 23:36:19.615332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.615307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bkhx" event={"ID":"a18d1704-9495-49e1-912d-b1da73e3da26","Type":"ContainerStarted","Data":"c14d5637a9647023ec372ee4d1514c6e6de0081254620024d64462e1b4eeb313"}
Sep 10 23:36:19.616777 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.616754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-spg86" event={"ID":"3d667255-226d-49be-a363-8cb6b3bc750d","Type":"ContainerStarted","Data":"e2e42efc926051e295c370286a90ea1641a7e1a73cb858ff74aa67f109c0f62d"}
Sep 10 23:36:19.621473 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.621434    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zmbws" podStartSLOduration=6.1813180899999995 podStartE2EDuration="20.621422583s" totalImagesPullingTime="14.440104493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:19.620889164 +0000 UTC m=+21.577120966" watchObservedRunningTime="2026-09-10 23:36:19.621422583 +0000 UTC m=+21.577654386"
Sep 10 23:36:19.636024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.635982    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-spg86" podStartSLOduration=3.62767074 podStartE2EDuration="20.635967285s" totalImagesPullingTime="17.008296545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:19.635860578 +0000 UTC m=+21.592092380" watchObservedRunningTime="2026-09-10 23:36:19.635967285 +0000 UTC m=+21.592199071"
Sep 10 23:36:19.701675 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.701619    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-88.ec2.internal" podStartSLOduration=19.70159898 podStartE2EDuration="19.70159898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:19.701520542 +0000 UTC m=+21.657752345" watchObservedRunningTime="2026-09-10 23:36:19.70159898 +0000 UTC m=+21.657830766"
Sep 10 23:36:19.702548 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.702508    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6bkhx" podStartSLOduration=3.700852779 podStartE2EDuration="20.702498054s" totalImagesPullingTime="17.001645275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:19.687880813 +0000 UTC m=+21.644112619" watchObservedRunningTime="2026-09-10 23:36:19.702498054 +0000 UTC m=+21.658729860"
Sep 10 23:36:19.720677 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.720638    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9fm6t" podStartSLOduration=2.237270813 podStartE2EDuration="20.720624528s" totalImagesPullingTime="18.483353715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:19.720402094 +0000 UTC m=+21.676633908" watchObservedRunningTime="2026-09-10 23:36:19.720624528 +0000 UTC m=+21.676856331"
Sep 10 23:36:19.736104 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:19.736044    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bfnfj" podStartSLOduration=6.288813212 podStartE2EDuration="20.736025738s" totalImagesPullingTime="14.447212526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:19.735373863 +0000 UTC m=+21.691605667" watchObservedRunningTime="2026-09-10 23:36:19.736025738 +0000 UTC m=+21.692257542"
Sep 10 23:36:20.528291 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.528243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:20.528470 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:20.528402    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:20.528972 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.528946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:20.529085 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:20.529049    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:20.529159 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.529141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:20.529287 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:20.529248    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:20.620625 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.620595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n7q2d" event={"ID":"d41773f0-a395-44dd-a28d-f8e3a3659411","Type":"ContainerStarted","Data":"61404f0c3648275ef31c53db7a438cfcc5852d1d382b7f45576abdccf88d8c25"}
Sep 10 23:36:20.625041 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.625012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"ad8765f4aa8ab6921f0def6de2138781d0bd714b55a6bdec63bc4862e3f1f886"}
Sep 10 23:36:20.625153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.625098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"a163a3f1c1d421b99817e5de3b77536dbee0ec3a462013bb9495dec42b9ce86d"}
Sep 10 23:36:20.625153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.625152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"413d8e4e7a131046a2f71c99793261eda9cf24291cc273d52e3c28aa2e0f77c6"}
Sep 10 23:36:20.625266 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.625199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"2a1cda1b74f7b7f0e9cfc078c64a05db63653eab2005c8f8004204fb67817177"}
Sep 10 23:36:20.625266 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.625247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"d0b2874119f40d4acb0cd7cbde3e1f1f7349394d4f289f006ef75cc9f125b25b"}
Sep 10 23:36:20.636207 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.636159    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n7q2d" podStartSLOduration=3.342194082 podStartE2EDuration="21.636141575s" totalImagesPullingTime="18.293947493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:20.636094019 +0000 UTC m=+22.592325825" watchObservedRunningTime="2026-09-10 23:36:20.636141575 +0000 UTC m=+22.592373379"
Sep 10 23:36:20.729109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:20.729083    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 23:36:21.109089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.109055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:21.109674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.109657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:21.528507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.528395    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:36:20.729103456Z","UUID":"1912c93a-2659-4383-93d4-2dd15cff2781","Handler":null,"Name":"","Endpoint":""}
Sep 10 23:36:21.531183 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.531086    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 23:36:21.531183 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.531138    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 23:36:21.629365 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.629306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g" event={"ID":"36b3a728-1f13-4576-963d-f0efea8f80d3","Type":"ContainerStarted","Data":"8437b7f919aa793aca1c964f15e310721035c7f7808cd8fe559a251e8aa349d3"}
Sep 10 23:36:21.629934 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.629605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:21.630197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:21.630179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6bkhx"
Sep 10 23:36:22.528467 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:22.528269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:22.528722 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:22.528565    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:22.528722 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:22.528269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:22.528722 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:22.528674    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:22.528722 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:22.528356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:22.528936 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:22.528838    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:22.633697 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:22.633658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g" event={"ID":"36b3a728-1f13-4576-963d-f0efea8f80d3","Type":"ContainerStarted","Data":"370f2b216730e17d6386bd2be13c51c84365d7d7c895d7b0bb4e36c606ce135c"}
Sep 10 23:36:22.637753 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:22.637728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"a94cc5d3afaca01f81f75c5a5fee1ccafcb402b4da02786c0862cdc60f0b2e4a"}
Sep 10 23:36:22.651219 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:22.651175    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p24g" podStartSLOduration=7.088237046 podStartE2EDuration="23.651157424s" totalImagesPullingTime="16.562920378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:22.649743678 +0000 UTC m=+24.605975481" watchObservedRunningTime="2026-09-10 23:36:22.651157424 +0000 UTC m=+24.607389228"
Sep 10 23:36:24.528651 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.528475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:24.529106 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.528476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:24.529106 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:24.528739    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:24.529106 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.528476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:24.529106 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:24.528764    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:24.529106 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:24.528895    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:24.643975 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.643918    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d" containerID="46106acede7f7e1a0502fb596f26ebd81ca0bcb0cfbae0f48f7c353fc8dce0b0" exitCode=0
Sep 10 23:36:24.644076 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.644005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerDied","Data":"46106acede7f7e1a0502fb596f26ebd81ca0bcb0cfbae0f48f7c353fc8dce0b0"}
Sep 10 23:36:24.647769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.647744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" event={"ID":"7dba2314-a579-4782-8272-e1af9b28a637","Type":"ContainerStarted","Data":"15a208d63c71e936be09e3d1d467c4311e1e3f1e110f82688e257d0a8ff71dda"}
Sep 10 23:36:24.648045 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.648030    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:24.648121 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.648100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:24.648156 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.648133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:24.662536 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.662515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:24.662759 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.662744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:24.693242 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:24.693191    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr" podStartSLOduration=8.475017694 podStartE2EDuration="25.693179557s" totalImagesPullingTime="17.218161863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:24.69250673 +0000 UTC m=+26.648738533" watchObservedRunningTime="2026-09-10 23:36:24.693179557 +0000 UTC m=+26.649411360"
Sep 10 23:36:26.045120 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.045085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5lhn7"]
Sep 10 23:36:26.045876 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.045202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:26.045876 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:26.045336    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:26.049013 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.048970    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z6rwl"]
Sep 10 23:36:26.049160 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.049081    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:26.049231 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:26.049194    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:26.049559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.049536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vk9qm"]
Sep 10 23:36:26.049671 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.049610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:26.049723 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:26.049683    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:26.656417 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.656383    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d" containerID="19960023e235f170c15e8cf0324512fe321081b4bdcc01c86213c5b1c82d2613" exitCode=0
Sep 10 23:36:26.656597 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:26.656463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerDied","Data":"19960023e235f170c15e8cf0324512fe321081b4bdcc01c86213c5b1c82d2613"}
Sep 10 23:36:27.528055 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:27.528026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:27.528546 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:27.528127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:27.528546 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:27.528153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:27.528546 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:27.528141    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:27.528546 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:27.528238    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:27.528546 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:27.528316    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:27.660934 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:27.660843    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d" containerID="6a2c40f0f47db912fcc062845a0c3f775cf9542e15f9353d5273992f5f3f2238" exitCode=0
Sep 10 23:36:27.660934 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:27.660903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerDied","Data":"6a2c40f0f47db912fcc062845a0c3f775cf9542e15f9353d5273992f5f3f2238"}
Sep 10 23:36:29.527977 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:29.527770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:29.528537 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:29.527822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:29.528537 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:29.528061    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:29.528537 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:29.528115    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:29.528537 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:29.527826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:29.528537 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:29.528186    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:31.528091 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:31.528057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:31.528091 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:31.528090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:31.528543 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:31.528181    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6rwl" podUID="4a62a794-f5a8-423e-bb4d-20ec9edccf4b"
Sep 10 23:36:31.528543 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:31.528222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:31.528543 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:31.528330    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5lhn7" podUID="5b828bf5-24ab-4b4c-8d09-2279c1c4c709"
Sep 10 23:36:31.528543 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:31.528399    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk9qm" podUID="7155a075-78c6-484c-9324-2f375d1ad2ac"
Sep 10 23:36:32.228430 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.228345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:32.228430 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.228394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:32.228664 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.228523    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:32.228664 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.228564    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:32.228664 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.228605    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.228584583 +0000 UTC m=+66.184816384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:32.228664 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.228623    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret podName:5b828bf5-24ab-4b4c-8d09-2279c1c4c709 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.228615919 +0000 UTC m=+66.184847906 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret") pod "global-pull-secret-syncer-5lhn7" (UID: "5b828bf5-24ab-4b4c-8d09-2279c1c4c709") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:32.379398 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.379370    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-88.ec2.internal" event="NodeReady"
Sep 10 23:36:32.379586 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.379533    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 23:36:32.412051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.412018    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"]
Sep 10 23:36:32.429965 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.429868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:32.430192 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.430052    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:32.430192 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.430078    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:32.430192 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.430091    2578 projected.go:196] Error preparing data for projected volume kube-api-access-k7br6 for pod openshift-network-diagnostics/network-check-target-vk9qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:32.430192 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.430152    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6 podName:7155a075-78c6-484c-9324-2f375d1ad2ac nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.430134266 +0000 UTC m=+66.386366050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7br6" (UniqueName: "kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6") pod "network-check-target-vk9qm" (UID: "7155a075-78c6-484c-9324-2f375d1ad2ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:32.436264 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.436240    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bzgp5"]
Sep 10 23:36:32.436414 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.436382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.439218 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.439064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 23:36:32.439218 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.439079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 23:36:32.439218 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.439069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vhrn5\""
Sep 10 23:36:32.439218 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.439172    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 23:36:32.452159 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.452137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 23:36:32.452335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.452290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"]
Sep 10 23:36:32.452444 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.452341    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v5jdk"]
Sep 10 23:36:32.452444 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.452362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.454742 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.454723    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 23:36:32.454880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.454860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9kf9v\""
Sep 10 23:36:32.455007 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.454986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 23:36:32.470955 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.470931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v5jdk"]
Sep 10 23:36:32.471068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.470965    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bzgp5"]
Sep 10 23:36:32.471068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.471002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:32.473701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.473677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 23:36:32.473805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.473677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 23:36:32.473805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.473681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 23:36:32.473805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.473717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-njwmd\""
Sep 10 23:36:32.530409 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-trusted-ca\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-installation-pull-secrets\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-bound-sa-token\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-tmp-dir\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg82z\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-kube-api-access-jg82z\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f4caf2f-2748-4852-afc6-19f7bf42c34e-ca-trust-extracted\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-certificates\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-image-registry-private-configuration\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.530805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8487\" (UniqueName: \"kubernetes.io/projected/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-kube-api-access-c8487\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.531239 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.530823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-config-volume\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.631917 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.631878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.631959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-tmp-dir\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.631986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg82z\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-kube-api-access-jg82z\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f4caf2f-2748-4852-afc6-19f7bf42c34e-ca-trust-extracted\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-certificates\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.632042    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.632067    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7f6bf58d-tvcbw: secret "image-registry-tls" not found
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880f83bd-a861-4ca0-9e53-b84389e50e8d-cert\") pod \"ingress-canary-v5jdk\" (UID: \"880f83bd-a861-4ca0-9e53-b84389e50e8d\") " pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:32.632103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-image-registry-private-configuration\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.632148    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls podName:2f4caf2f-2748-4852-afc6-19f7bf42c34e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:33.132125903 +0000 UTC m=+35.088357709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls") pod "image-registry-6b7f6bf58d-tvcbw" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e") : secret "image-registry-tls" not found
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdktc\" (UniqueName: \"kubernetes.io/projected/880f83bd-a861-4ca0-9e53-b84389e50e8d-kube-api-access-rdktc\") pod \"ingress-canary-v5jdk\" (UID: \"880f83bd-a861-4ca0-9e53-b84389e50e8d\") " pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8487\" (UniqueName: \"kubernetes.io/projected/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-kube-api-access-c8487\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-config-volume\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-trusted-ca\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-installation-pull-secrets\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-tmp-dir\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.632508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.632403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-bound-sa-token\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.634032 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.633104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-config-volume\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.634032 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.633117    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:36:32.634032 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:32.633179    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls podName:f1c2427c-a1e9-4e53-a2de-e7b8630507fe nodeName:}" failed. No retries permitted until 2026-09-10 23:36:33.133163711 +0000 UTC m=+35.089395505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls") pod "dns-default-bzgp5" (UID: "f1c2427c-a1e9-4e53-a2de-e7b8630507fe") : secret "dns-default-metrics-tls" not found
Sep 10 23:36:32.634032 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.633392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-certificates\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.634032 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.633508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f4caf2f-2748-4852-afc6-19f7bf42c34e-ca-trust-extracted\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.634032 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.633993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-trusted-ca\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.636589 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.636564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-installation-pull-secrets\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.636589 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.636576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-image-registry-private-configuration\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.640637 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.640587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8487\" (UniqueName: \"kubernetes.io/projected/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-kube-api-access-c8487\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:32.641049 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.641027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-bound-sa-token\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.641418 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.641402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg82z\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-kube-api-access-jg82z\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:32.733175 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.733147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880f83bd-a861-4ca0-9e53-b84389e50e8d-cert\") pod \"ingress-canary-v5jdk\" (UID: \"880f83bd-a861-4ca0-9e53-b84389e50e8d\") " pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:32.733337 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.733182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdktc\" (UniqueName: \"kubernetes.io/projected/880f83bd-a861-4ca0-9e53-b84389e50e8d-kube-api-access-rdktc\") pod \"ingress-canary-v5jdk\" (UID: \"880f83bd-a861-4ca0-9e53-b84389e50e8d\") " pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:32.735262 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.735240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880f83bd-a861-4ca0-9e53-b84389e50e8d-cert\") pod \"ingress-canary-v5jdk\" (UID: \"880f83bd-a861-4ca0-9e53-b84389e50e8d\") " pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:32.740710 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.740689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdktc\" (UniqueName: \"kubernetes.io/projected/880f83bd-a861-4ca0-9e53-b84389e50e8d-kube-api-access-rdktc\") pod \"ingress-canary-v5jdk\" (UID: \"880f83bd-a861-4ca0-9e53-b84389e50e8d\") " pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:32.780920 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:32.780860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v5jdk"
Sep 10 23:36:33.080154 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.080004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v5jdk"]
Sep 10 23:36:33.091234 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:36:33.091214    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880f83bd_a861_4ca0_9e53_b84389e50e8d.slice/crio-c025a20cf426a12c4e58053b21a60ae4a6617df34dadef4f356ccb4cae6be34e WatchSource:0}: Error finding container c025a20cf426a12c4e58053b21a60ae4a6617df34dadef4f356ccb4cae6be34e: Status 404 returned error can't find the container with id c025a20cf426a12c4e58053b21a60ae4a6617df34dadef4f356ccb4cae6be34e
Sep 10 23:36:33.136372 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.136347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:33.136495 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.136401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:33.136562 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:33.136502    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:36:33.136562 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:33.136518    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7f6bf58d-tvcbw: secret "image-registry-tls" not found
Sep 10 23:36:33.136562 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:33.136558    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls podName:2f4caf2f-2748-4852-afc6-19f7bf42c34e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.136544796 +0000 UTC m=+36.092776577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls") pod "image-registry-6b7f6bf58d-tvcbw" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e") : secret "image-registry-tls" not found
Sep 10 23:36:33.136731 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:33.136500    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:36:33.136731 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:33.136644    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls podName:f1c2427c-a1e9-4e53-a2de-e7b8630507fe nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.136625727 +0000 UTC m=+36.092857511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls") pod "dns-default-bzgp5" (UID: "f1c2427c-a1e9-4e53-a2de-e7b8630507fe") : secret "dns-default-metrics-tls" not found
Sep 10 23:36:33.528045 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.528015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:36:33.528211 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.528113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:36:33.528211 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.528197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:36:33.530718 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.530695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:36:33.531217 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.530821    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:36:33.531217 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.530916    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:36:33.531917 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.531897    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:36:33.532038 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.531892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xmh4\""
Sep 10 23:36:33.532038 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.531900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t4h6l\""
Sep 10 23:36:33.678477 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.678443    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d" containerID="d230ad5f83031fdbd8c2da97035d0ff1c58cfcd94cf1fd78aaae157bf470f6b4" exitCode=0
Sep 10 23:36:33.678672 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.678527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerDied","Data":"d230ad5f83031fdbd8c2da97035d0ff1c58cfcd94cf1fd78aaae157bf470f6b4"}
Sep 10 23:36:33.680169 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:33.680145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v5jdk" event={"ID":"880f83bd-a861-4ca0-9e53-b84389e50e8d","Type":"ContainerStarted","Data":"c025a20cf426a12c4e58053b21a60ae4a6617df34dadef4f356ccb4cae6be34e"}
Sep 10 23:36:34.006588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.006550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:34.127443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.127359    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:34.144561 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.144524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:34.144704 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:34.144638    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:36:34.144704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.144653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:34.144824 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:34.144663    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7f6bf58d-tvcbw: secret "image-registry-tls" not found
Sep 10 23:36:34.144824 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:34.144705    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:36:34.144824 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:34.144783    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls podName:2f4caf2f-2748-4852-afc6-19f7bf42c34e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:36.144767891 +0000 UTC m=+38.100999671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls") pod "image-registry-6b7f6bf58d-tvcbw" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e") : secret "image-registry-tls" not found
Sep 10 23:36:34.144968 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:34.144831    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls podName:f1c2427c-a1e9-4e53-a2de-e7b8630507fe nodeName:}" failed. No retries permitted until 2026-09-10 23:36:36.144811902 +0000 UTC m=+38.101043698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls") pod "dns-default-bzgp5" (UID: "f1c2427c-a1e9-4e53-a2de-e7b8630507fe") : secret "dns-default-metrics-tls" not found
Sep 10 23:36:34.203650 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.203605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:34.303418 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.303386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:34.369876 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.369840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:34.686208 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.686171    2578 generic.go:356] "Generic (PLEG): container finished" podID="4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d" containerID="d1cf78ba29cdf3fada5d330d7071d7655e21c62aef52c541885885e23aaf5277" exitCode=0
Sep 10 23:36:34.686876 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.686236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerDied","Data":"d1cf78ba29cdf3fada5d330d7071d7655e21c62aef52c541885885e23aaf5277"}
Sep 10 23:36:34.980079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:34.979905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:35.080184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:35.080147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:35.374787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:35.374751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:35.477300 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:35.476915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:35.691686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:35.691661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" event={"ID":"4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d","Type":"ContainerStarted","Data":"fa1723b140062f9e33041337ff3910ca35f1c2eec171f629bce33ee33dc662f9"}
Sep 10 23:36:35.720132 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:35.720091    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qk2k7" podStartSLOduration=8.049354342 podStartE2EDuration="36.72007433s" totalImagesPullingTime="28.670719988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:35.718741376 +0000 UTC m=+37.674973179" watchObservedRunningTime="2026-09-10 23:36:35.72007433 +0000 UTC m=+37.676306133"
Sep 10 23:36:36.161339 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.161304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:36.161479 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.161392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:36.161479 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:36.161457    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:36:36.161479 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:36.161472    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:36:36.161479 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:36.161477    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7f6bf58d-tvcbw: secret "image-registry-tls" not found
Sep 10 23:36:36.161602 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:36.161529    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls podName:f1c2427c-a1e9-4e53-a2de-e7b8630507fe nodeName:}" failed. No retries permitted until 2026-09-10 23:36:40.16151492 +0000 UTC m=+42.117746716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls") pod "dns-default-bzgp5" (UID: "f1c2427c-a1e9-4e53-a2de-e7b8630507fe") : secret "dns-default-metrics-tls" not found
Sep 10 23:36:36.161602 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:36.161543    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls podName:2f4caf2f-2748-4852-afc6-19f7bf42c34e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:40.161537832 +0000 UTC m=+42.117769613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls") pod "image-registry-6b7f6bf58d-tvcbw" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e") : secret "image-registry-tls" not found
Sep 10 23:36:36.171065 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.171035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:36.272378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.272342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:36.571004 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.570968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:36.626707 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.626675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:36.695157 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.695125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v5jdk" event={"ID":"880f83bd-a861-4ca0-9e53-b84389e50e8d","Type":"ContainerStarted","Data":"4e83bfe4885f2caea4da550fe92d7249d5ea363f85fee97ec000e4561e573e03"}
Sep 10 23:36:36.727664 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:36.727613    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v5jdk" podStartSLOduration=2.165080457 podStartE2EDuration="4.727598864s" totalImagesPullingTime="2.562518407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:36.727157875 +0000 UTC m=+38.683389692" watchObservedRunningTime="2026-09-10 23:36:36.727598864 +0000 UTC m=+38.683830667"
Sep 10 23:36:37.100957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:37.100926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:37.201959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:37.201926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:40.190331 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:40.190296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:40.190751 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:40.190375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:40.190751 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:40.190431    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:36:40.190751 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:40.190451    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7f6bf58d-tvcbw: secret "image-registry-tls" not found
Sep 10 23:36:40.190751 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:40.190484    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:36:40.190751 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:40.190507    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls podName:2f4caf2f-2748-4852-afc6-19f7bf42c34e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:48.190489866 +0000 UTC m=+50.146721646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls") pod "image-registry-6b7f6bf58d-tvcbw" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e") : secret "image-registry-tls" not found
Sep 10 23:36:40.190751 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:40.190531    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls podName:f1c2427c-a1e9-4e53-a2de-e7b8630507fe nodeName:}" failed. No retries permitted until 2026-09-10 23:36:48.190516929 +0000 UTC m=+50.146748714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls") pod "dns-default-bzgp5" (UID: "f1c2427c-a1e9-4e53-a2de-e7b8630507fe") : secret "dns-default-metrics-tls" not found
Sep 10 23:36:43.242052 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:43.242013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:43.298520 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:43.298482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:45.780908 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:45.780870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:45.829486 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:45.829448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:46.778552 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:46.778516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:46.837251 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:46.837217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:48.244507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:48.244467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:36:48.244912 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:48.244527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:36:48.244912 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:48.244614    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:36:48.244912 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:48.244620    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:36:48.244912 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:48.244640    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7f6bf58d-tvcbw: secret "image-registry-tls" not found
Sep 10 23:36:48.244912 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:48.244672    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls podName:f1c2427c-a1e9-4e53-a2de-e7b8630507fe nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.244659094 +0000 UTC m=+66.200890875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls") pod "dns-default-bzgp5" (UID: "f1c2427c-a1e9-4e53-a2de-e7b8630507fe") : secret "dns-default-metrics-tls" not found
Sep 10 23:36:48.244912 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:36:48.244696    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls podName:2f4caf2f-2748-4852-afc6-19f7bf42c34e nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.244682889 +0000 UTC m=+66.200914670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls") pod "image-registry-6b7f6bf58d-tvcbw" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e") : secret "image-registry-tls" not found
Sep 10 23:36:49.729862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:49.729827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:54.829213 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:54.829180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:54.919643 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:54.919598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:56.666536 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:56.666513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8rhr"
Sep 10 23:36:56.780415 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:56.780388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:56.875039 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:56.875011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:56.975584 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:56.975516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:57.028671 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:36:57.028647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:01.864567 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:01.864527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:01.961116 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:01.961078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:02.226861 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:02.226785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:03.200647 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.200611    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"]
Sep 10 23:37:03.206994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.206972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-sq4f8"]
Sep 10 23:37:03.207119 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.207101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.210109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.210078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.210481 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.210459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 23:37:03.210664 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.210642    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ssv8b\""
Sep 10 23:37:03.210790 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.210737    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 23:37:03.210790 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.210511    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.212176 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.212156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.214832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.214611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 23:37:03.214973 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.214953    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 23:37:03.215129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.215115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.215387 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.215372    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.215874 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.215846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"]
Sep 10 23:37:03.216291 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.216257    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n8v9t\""
Sep 10 23:37:03.220672 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.220652    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 23:37:03.232481 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.232463    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-sq4f8"]
Sep 10 23:37:03.244454 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.244433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvx4p\" (UniqueName: \"kubernetes.io/projected/01d9457a-ff0d-456e-b864-3204f0f384f0-kube-api-access-tvx4p\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.244548 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.244472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a008749-a748-49c8-abc4-a0506f8f2e69-config\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.244548 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.244519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbldm\" (UniqueName: \"kubernetes.io/projected/2a008749-a748-49c8-abc4-a0506f8f2e69-kube-api-access-hbldm\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.244548 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.244542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01d9457a-ff0d-456e-b864-3204f0f384f0-serving-cert\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.244673 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.244571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a008749-a748-49c8-abc4-a0506f8f2e69-trusted-ca\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.244673 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.244600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01d9457a-ff0d-456e-b864-3204f0f384f0-config\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.244749 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.244676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a008749-a748-49c8-abc4-a0506f8f2e69-serving-cert\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.296833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.296808    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"]
Sep 10 23:37:03.300800 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.300784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:03.303432 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.303406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 23:37:03.303539 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.303452    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8pf96\""
Sep 10 23:37:03.303539 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.303476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.303689 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.303589    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.310380 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.310359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"]
Sep 10 23:37:03.345617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a008749-a748-49c8-abc4-a0506f8f2e69-trusted-ca\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.345704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01d9457a-ff0d-456e-b864-3204f0f384f0-config\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.345704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:03.345815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a008749-a748-49c8-abc4-a0506f8f2e69-serving-cert\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.345815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvx4p\" (UniqueName: \"kubernetes.io/projected/01d9457a-ff0d-456e-b864-3204f0f384f0-kube-api-access-tvx4p\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.345875 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a008749-a748-49c8-abc4-a0506f8f2e69-config\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.345875 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbldm\" (UniqueName: \"kubernetes.io/projected/2a008749-a748-49c8-abc4-a0506f8f2e69-kube-api-access-hbldm\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.345960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01d9457a-ff0d-456e-b864-3204f0f384f0-serving-cert\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.345960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.345929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8gz\" (UniqueName: \"kubernetes.io/projected/74971d3e-ba66-4e20-8467-6aaba78f997d-kube-api-access-tr8gz\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:03.346264 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.346237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01d9457a-ff0d-456e-b864-3204f0f384f0-config\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.346492 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.346473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a008749-a748-49c8-abc4-a0506f8f2e69-config\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.346629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.346614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a008749-a748-49c8-abc4-a0506f8f2e69-trusted-ca\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.348938 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.348916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01d9457a-ff0d-456e-b864-3204f0f384f0-serving-cert\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.349110 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.349090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a008749-a748-49c8-abc4-a0506f8f2e69-serving-cert\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.353972 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.353952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbldm\" (UniqueName: \"kubernetes.io/projected/2a008749-a748-49c8-abc4-a0506f8f2e69-kube-api-access-hbldm\") pod \"console-operator-696884fb9c-sq4f8\" (UID: \"2a008749-a748-49c8-abc4-a0506f8f2e69\") " pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.354129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.354107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvx4p\" (UniqueName: \"kubernetes.io/projected/01d9457a-ff0d-456e-b864-3204f0f384f0-kube-api-access-tvx4p\") pod \"service-ca-operator-cf7c776b7-vdrjc\" (UID: \"01d9457a-ff0d-456e-b864-3204f0f384f0\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.405390 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.405359    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84c49bb84b-8nk86"]
Sep 10 23:37:03.410631 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.410612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.413329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.413308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 23:37:03.413482 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.413457    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 23:37:03.413722 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.413704    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 23:37:03.413796 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.413744    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.413796 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.413748    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dj424\""
Sep 10 23:37:03.413906 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.413880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 23:37:03.413986 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.413971    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.420836 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.420811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84c49bb84b-8nk86"]
Sep 10 23:37:03.446729 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.446707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.446812 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.446731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.446812 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.446768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8gz\" (UniqueName: \"kubernetes.io/projected/74971d3e-ba66-4e20-8467-6aaba78f997d-kube-api-access-tr8gz\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:03.446885 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.446849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-default-certificate\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.446919 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.446880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:03.446919 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.446899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-stats-auth\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.446992 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.446927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqf9\" (UniqueName: \"kubernetes.io/projected/4cfb42b9-bb91-4012-879e-b185d1cce2eb-kube-api-access-2gqf9\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.447030 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:03.446992    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:03.447066 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:03.447046    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls podName:74971d3e-ba66-4e20-8467-6aaba78f997d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:03.947032557 +0000 UTC m=+65.903264343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2jkv7" (UID: "74971d3e-ba66-4e20-8467-6aaba78f997d") : secret "samples-operator-tls" not found
Sep 10 23:37:03.455222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.455176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8gz\" (UniqueName: \"kubernetes.io/projected/74971d3e-ba66-4e20-8467-6aaba78f997d-kube-api-access-tr8gz\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:03.520470 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.520438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"
Sep 10 23:37:03.525041 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.525017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:03.538957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.538933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:03.547535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.547476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.547535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.547512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.547719 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.547589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-default-certificate\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.547719 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:03.547625    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.047604315 +0000 UTC m=+66.003836120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:03.547719 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:03.547679    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:03.547884 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:03.547723    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.047710272 +0000 UTC m=+66.003942070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : secret "router-metrics-certs-default" not found
Sep 10 23:37:03.547884 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.547676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-stats-auth\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.547884 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.547769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqf9\" (UniqueName: \"kubernetes.io/projected/4cfb42b9-bb91-4012-879e-b185d1cce2eb-kube-api-access-2gqf9\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.550312 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.550268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-stats-auth\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.550442 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.550268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-default-certificate\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.564681 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.564658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqf9\" (UniqueName: \"kubernetes.io/projected/4cfb42b9-bb91-4012-879e-b185d1cce2eb-kube-api-access-2gqf9\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:03.632725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.632695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:03.735545 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.735481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:03.793857 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.793830    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc"]
Sep 10 23:37:03.796177 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.796158    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-sq4f8"]
Sep 10 23:37:03.797026 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:03.797001    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d9457a_ff0d_456e_b864_3204f0f384f0.slice/crio-af11843bc2041be643b3a39e0ee6daa60eeb33af7d13612a45374a6dd57be28c WatchSource:0}: Error finding container af11843bc2041be643b3a39e0ee6daa60eeb33af7d13612a45374a6dd57be28c: Status 404 returned error can't find the container with id af11843bc2041be643b3a39e0ee6daa60eeb33af7d13612a45374a6dd57be28c
Sep 10 23:37:03.798124 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:03.798101    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a008749_a748_49c8_abc4_a0506f8f2e69.slice/crio-31d2acf02ec57ac8ef799c488e7c4db11e5e6b2916196e9fb3a79ae26face9b1 WatchSource:0}: Error finding container 31d2acf02ec57ac8ef799c488e7c4db11e5e6b2916196e9fb3a79ae26face9b1: Status 404 returned error can't find the container with id 31d2acf02ec57ac8ef799c488e7c4db11e5e6b2916196e9fb3a79ae26face9b1
Sep 10 23:37:03.804844 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.804812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:03.951610 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:03.951578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:03.951766 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:03.951731    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:03.951806 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:03.951795    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls podName:74971d3e-ba66-4e20-8467-6aaba78f997d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.951781565 +0000 UTC m=+66.908013346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2jkv7" (UID: "74971d3e-ba66-4e20-8467-6aaba78f997d") : secret "samples-operator-tls" not found
Sep 10 23:37:04.010545 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.010473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:04.052611 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.052589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:04.052726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.052634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:04.052765 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.052750    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:04.052798 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.052762    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.052745228 +0000 UTC m=+67.008977026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:04.052798 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.052786    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.052776705 +0000 UTC m=+67.009008487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : secret "router-metrics-certs-default" not found
Sep 10 23:37:04.111448 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.111418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:04.253438 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.253394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.253468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.253487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.253502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.253572    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.253625    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.253646    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7f6bf58d-tvcbw: secret "image-registry-tls" not found
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.253634    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls podName:f1c2427c-a1e9-4e53-a2de-e7b8630507fe nodeName:}" failed. No retries permitted until 2026-09-10 23:37:36.253618074 +0000 UTC m=+98.209849859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls") pod "dns-default-bzgp5" (UID: "f1c2427c-a1e9-4e53-a2de-e7b8630507fe") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:04.253826 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.253700    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls podName:2f4caf2f-2748-4852-afc6-19f7bf42c34e nodeName:}" failed. No retries permitted until 2026-09-10 23:37:36.253684118 +0000 UTC m=+98.209915899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls") pod "image-registry-6b7f6bf58d-tvcbw" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e") : secret "image-registry-tls" not found
Sep 10 23:37:04.256299 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.256267    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:37:04.256608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.256595    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:37:04.264014 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.263998    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 23:37:04.264072 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.264035    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs podName:4a62a794-f5a8-423e-bb4d-20ec9edccf4b nodeName:}" failed. No retries permitted until 2026-09-10 23:38:08.264026044 +0000 UTC m=+130.220257830 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs") pod "network-metrics-daemon-z6rwl" (UID: "4a62a794-f5a8-423e-bb4d-20ec9edccf4b") : secret "metrics-daemon-secret" not found
Sep 10 23:37:04.265928 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.265913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b828bf5-24ab-4b4c-8d09-2279c1c4c709-original-pull-secret\") pod \"global-pull-secret-syncer-5lhn7\" (UID: \"5b828bf5-24ab-4b4c-8d09-2279c1c4c709\") " pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:37:04.447604 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.447560    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5lhn7"
Sep 10 23:37:04.454880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.454844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:37:04.457711 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.457688    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:37:04.467959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.467720    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:37:04.479959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.479928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7br6\" (UniqueName: \"kubernetes.io/projected/7155a075-78c6-484c-9324-2f375d1ad2ac-kube-api-access-k7br6\") pod \"network-check-target-vk9qm\" (UID: \"7155a075-78c6-484c-9324-2f375d1ad2ac\") " pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:37:04.732268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.732057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5lhn7"]
Sep 10 23:37:04.734818 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:04.734789    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b828bf5_24ab_4b4c_8d09_2279c1c4c709.slice/crio-fa15d3a18277d584759b9646580e4fd85a0ced792fc5f2a5bbfda23a68651ef0 WatchSource:0}: Error finding container fa15d3a18277d584759b9646580e4fd85a0ced792fc5f2a5bbfda23a68651ef0: Status 404 returned error can't find the container with id fa15d3a18277d584759b9646580e4fd85a0ced792fc5f2a5bbfda23a68651ef0
Sep 10 23:37:04.740587 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.740521    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t4h6l\""
Sep 10 23:37:04.749038 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.748957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:37:04.758952 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.758892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5lhn7" event={"ID":"5b828bf5-24ab-4b4c-8d09-2279c1c4c709","Type":"ContainerStarted","Data":"fa15d3a18277d584759b9646580e4fd85a0ced792fc5f2a5bbfda23a68651ef0"}
Sep 10 23:37:04.760297 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.760224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc" event={"ID":"01d9457a-ff0d-456e-b864-3204f0f384f0","Type":"ContainerStarted","Data":"af11843bc2041be643b3a39e0ee6daa60eeb33af7d13612a45374a6dd57be28c"}
Sep 10 23:37:04.761906 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.761858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" event={"ID":"2a008749-a748-49c8-abc4-a0506f8f2e69","Type":"ContainerStarted","Data":"31d2acf02ec57ac8ef799c488e7c4db11e5e6b2916196e9fb3a79ae26face9b1"}
Sep 10 23:37:04.959531 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:04.959489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:04.959709 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.959657    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:04.959771 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:04.959734    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls podName:74971d3e-ba66-4e20-8467-6aaba78f997d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:06.959713702 +0000 UTC m=+68.915945484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2jkv7" (UID: "74971d3e-ba66-4e20-8467-6aaba78f997d") : secret "samples-operator-tls" not found
Sep 10 23:37:05.043322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:05.043287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vk9qm"]
Sep 10 23:37:05.046120 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:05.046083    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7155a075_78c6_484c_9324_2f375d1ad2ac.slice/crio-ebbfec16953de6aa58cba74c14fab5f931690d2a27fe5a14e53ba5a441e23d1b WatchSource:0}: Error finding container ebbfec16953de6aa58cba74c14fab5f931690d2a27fe5a14e53ba5a441e23d1b: Status 404 returned error can't find the container with id ebbfec16953de6aa58cba74c14fab5f931690d2a27fe5a14e53ba5a441e23d1b
Sep 10 23:37:05.060611 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:05.060582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:05.060730 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:05.060625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:05.060864 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:05.060847    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:05.060928 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:05.060872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.060849039 +0000 UTC m=+69.017080837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:05.060928 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:05.060901    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.060885043 +0000 UTC m=+69.017116829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : secret "router-metrics-certs-default" not found
Sep 10 23:37:05.766061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:05.766018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vk9qm" event={"ID":"7155a075-78c6-484c-9324-2f375d1ad2ac","Type":"ContainerStarted","Data":"ebbfec16953de6aa58cba74c14fab5f931690d2a27fe5a14e53ba5a441e23d1b"}
Sep 10 23:37:06.771905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:06.771770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc" event={"ID":"01d9457a-ff0d-456e-b864-3204f0f384f0","Type":"ContainerStarted","Data":"b122cdb03fc99254853fa6d3b545c895443acfca721e77f85144e1186754e97f"}
Sep 10 23:37:06.775417 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:06.775389    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/0.log"
Sep 10 23:37:06.775532 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:06.775436    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a008749-a748-49c8-abc4-a0506f8f2e69" containerID="2885893d20bdbcebdbeb34d93709259de7c899e68100c44f7c5b86ee3e2410e0" exitCode=255
Sep 10 23:37:06.775532 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:06.775482    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" event={"ID":"2a008749-a748-49c8-abc4-a0506f8f2e69","Type":"ContainerDied","Data":"2885893d20bdbcebdbeb34d93709259de7c899e68100c44f7c5b86ee3e2410e0"}
Sep 10 23:37:06.775799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:06.775781    2578 scope.go:117] "RemoveContainer" containerID="2885893d20bdbcebdbeb34d93709259de7c899e68100c44f7c5b86ee3e2410e0"
Sep 10 23:37:06.788921 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:06.788867    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc" podStartSLOduration=1.126806368 podStartE2EDuration="3.788847118s" totalImagesPullingTime="2.66204075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:06.787900721 +0000 UTC m=+68.744132525" watchObservedRunningTime="2026-09-10 23:37:06.788847118 +0000 UTC m=+68.745078924"
Sep 10 23:37:06.974899 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:06.974870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:06.975060 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:06.975032    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:06.975142 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:06.975108    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls podName:74971d3e-ba66-4e20-8467-6aaba78f997d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:10.975088446 +0000 UTC m=+72.931320228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2jkv7" (UID: "74971d3e-ba66-4e20-8467-6aaba78f997d") : secret "samples-operator-tls" not found
Sep 10 23:37:07.076068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.075994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:07.076068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.076042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:07.076266 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:07.076219    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:07.076354 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:07.076296    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.076260912 +0000 UTC m=+73.032492701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : secret "router-metrics-certs-default" not found
Sep 10 23:37:07.076417 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:07.076369    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.076359049 +0000 UTC m=+73.032590833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:07.214772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.214736    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:07.317434 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.317358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:07.781097 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.781061    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/1.log"
Sep 10 23:37:07.781804 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.781765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/0.log"
Sep 10 23:37:07.781921 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.781815    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a008749-a748-49c8-abc4-a0506f8f2e69" containerID="2732e9784d8cade5661443ed477b2b0f6c7281dcd1ccf45c44618cfa8502e387" exitCode=255
Sep 10 23:37:07.781988 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.781912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" event={"ID":"2a008749-a748-49c8-abc4-a0506f8f2e69","Type":"ContainerDied","Data":"2732e9784d8cade5661443ed477b2b0f6c7281dcd1ccf45c44618cfa8502e387"}
Sep 10 23:37:07.781988 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.781978    2578 scope.go:117] "RemoveContainer" containerID="2885893d20bdbcebdbeb34d93709259de7c899e68100c44f7c5b86ee3e2410e0"
Sep 10 23:37:07.782179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:07.782163    2578 scope.go:117] "RemoveContainer" containerID="2732e9784d8cade5661443ed477b2b0f6c7281dcd1ccf45c44618cfa8502e387"
Sep 10 23:37:07.782384 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:07.782362    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-sq4f8_openshift-console-operator(2a008749-a748-49c8-abc4-a0506f8f2e69)\"" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" podUID="2a008749-a748-49c8-abc4-a0506f8f2e69"
Sep 10 23:37:08.243598 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:08.243504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:08.300079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:08.298925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:08.680240 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:08.680210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:08.772419 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:08.772374    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:08.784976 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:08.784953    2578 scope.go:117] "RemoveContainer" containerID="2732e9784d8cade5661443ed477b2b0f6c7281dcd1ccf45c44618cfa8502e387"
Sep 10 23:37:08.785432 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:08.785165    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-sq4f8_openshift-console-operator(2a008749-a748-49c8-abc4-a0506f8f2e69)\"" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" podUID="2a008749-a748-49c8-abc4-a0506f8f2e69"
Sep 10 23:37:08.867192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:08.867159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:08.968389 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:08.968246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:09.021089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:09.021045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:09.851487 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:09.851396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:09.945640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:09.945605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:10.047059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.047032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:10.624030 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.623997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:10.791685 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.791649    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/1.log"
Sep 10 23:37:10.793179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.793152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5lhn7" event={"ID":"5b828bf5-24ab-4b4c-8d09-2279c1c4c709","Type":"ContainerStarted","Data":"fd058d0105c7705787321799c752ba125bb16169c78e7d6259828af939217531"}
Sep 10 23:37:10.794539 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.794514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vk9qm" event={"ID":"7155a075-78c6-484c-9324-2f375d1ad2ac","Type":"ContainerStarted","Data":"3ac2930a888577aaa9ef4b0a0c12d7675bc6b5425f7fd68e34f6aef231334cdf"}
Sep 10 23:37:10.794661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.794645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:37:10.816078 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.816036    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5lhn7" podStartSLOduration=66.565657241 podStartE2EDuration="1m11.816021558s" totalImagesPullingTime="5.250364317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.814330235 +0000 UTC m=+72.770562038" watchObservedRunningTime="2026-09-10 23:37:10.816021558 +0000 UTC m=+72.772253361"
Sep 10 23:37:10.842086 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.842044    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vk9qm" podStartSLOduration=65.904072985 podStartE2EDuration="1m10.842030777s" totalImagesPullingTime="4.937957792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.841182806 +0000 UTC m=+72.797414645" watchObservedRunningTime="2026-09-10 23:37:10.842030777 +0000 UTC m=+72.798262580"
Sep 10 23:37:10.870081 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.870057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:10.881834 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.881780    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hrvpr"]
Sep 10 23:37:10.884910 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.884883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:10.890456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.890421    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 23:37:10.890456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.890441    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 23:37:10.890600 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.890546    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 23:37:10.890656 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.890620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-424q2\""
Sep 10 23:37:10.890656 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.890646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 23:37:10.898704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.898685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hrvpr"]
Sep 10 23:37:10.911948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.911928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-crio-socket\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:10.912040 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.912018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-data-volume\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:10.912103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.912047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:10.912103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.912068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4szv\" (UniqueName: \"kubernetes.io/projected/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-api-access-h4szv\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:10.912205 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.912183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:10.983053 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:10.983023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:11.012958 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.012934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-crio-socket\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.013059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.012986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:11.013059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.013042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-crio-socket\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.013163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.013044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-data-volume\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.013163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.013084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.013163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.013107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4szv\" (UniqueName: \"kubernetes.io/projected/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-api-access-h4szv\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.013163 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.013136    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:11.013163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.013150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.013437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.013152    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:11.013437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.013194    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.513176228 +0000 UTC m=+73.469408033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:11.013437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.013220    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls podName:74971d3e-ba66-4e20-8467-6aaba78f997d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.013204987 +0000 UTC m=+80.969436782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2jkv7" (UID: "74971d3e-ba66-4e20-8467-6aaba78f997d") : secret "samples-operator-tls" not found
Sep 10 23:37:11.013437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.013250    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:11.013437 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.013313    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.513299912 +0000 UTC m=+73.469531705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:11.013437 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.013372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-data-volume\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.022394 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.022371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4szv\" (UniqueName: \"kubernetes.io/projected/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-api-access-h4szv\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.113624 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.113595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:11.113733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.113635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:11.113773 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.113729    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:11.113807 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.113771    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.113756839 +0000 UTC m=+81.069988622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:11.113807 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.113786    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.113780221 +0000 UTC m=+81.070012001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : secret "router-metrics-certs-default" not found
Sep 10 23:37:11.516813 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.516761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.517024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.516870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:11.517024 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.516934    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:11.517024 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.516938    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:11.517024 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.516983    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:12.516970181 +0000 UTC m=+74.473201962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:11.517024 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:11.516996    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:12.516989736 +0000 UTC m=+74.473221517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:11.555596 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:11.555571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:12.279220 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:12.279192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:12.340733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:12.340694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:12.525219 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:12.525178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:12.525406 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:12.525229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:12.525406 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:12.525316    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:12.525406 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:12.525355    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:12.525406 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:12.525377    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.525363141 +0000 UTC m=+76.481594922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:12.525406 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:12.525397    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.525385989 +0000 UTC m=+76.481617769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:13.421233 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:13.421205    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bfnfj_8ae7c2aa-ebd2-4925-a132-c5b91088a016/dns-node-resolver/0.log"
Sep 10 23:37:13.526092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:13.526055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:13.526251 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:13.526110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:13.526468 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:13.526455    2578 scope.go:117] "RemoveContainer" containerID="2732e9784d8cade5661443ed477b2b0f6c7281dcd1ccf45c44618cfa8502e387"
Sep 10 23:37:13.526619 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:13.526600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-sq4f8_openshift-console-operator(2a008749-a748-49c8-abc4-a0506f8f2e69)\"" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" podUID="2a008749-a748-49c8-abc4-a0506f8f2e69"
Sep 10 23:37:14.401138 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.401102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.501263 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.501228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.541632 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.541605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:14.541764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.541651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:14.541764 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:14.541723    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.541764 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:14.541737    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.541875 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:14.541783    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:18.541767448 +0000 UTC m=+80.497999229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.541875 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:14.541800    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:18.541791414 +0000 UTC m=+80.498023200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.637705 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.637675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.777529 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.777495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.823359 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.823332    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmbws_453364b7-bff4-4640-8fb2-07ba8b5f95c9/node-ca/0.log"
Sep 10 23:37:14.875702 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.875660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.995201 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:14.995166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:15.110358 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:15.110264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:15.420736 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:15.420664    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v5jdk_880f83bd-a861-4ca0-9e53-b84389e50e8d/serve-healthcheck-canary/0.log"
Sep 10 23:37:15.901682 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:15.901646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:16.094974 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:16.094915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:16.689538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:16.689495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:17.098029 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:17.097994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:17.487845 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:17.487736    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:17.734668 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:17.734631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:17.908523 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:17.908490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:18.095473 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:18.095441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:18.198462 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:18.198376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:18.399727 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:18.399692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:18.577685 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:18.577657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:18.577874 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:18.577704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:18.577874 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:18.577785    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:18.577874 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:18.577816    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:18.577874 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:18.577854    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:26.577834581 +0000 UTC m=+88.534066380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:18.577874 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:18.577872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls podName:36fe9cad-01bc-41f0-8718-85cd9b5ba32d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:26.577863779 +0000 UTC m=+88.534095561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrvpr" (UID: "36fe9cad-01bc-41f0-8718-85cd9b5ba32d") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:18.903026 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:18.902937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.081433 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.081397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:19.083808 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.083784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74971d3e-ba66-4e20-8467-6aaba78f997d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2jkv7\" (UID: \"74971d3e-ba66-4e20-8467-6aaba78f997d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:19.087982 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.087951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.154024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.153955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.182570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.182545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:19.182639 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.182579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:19.182774 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:19.182762    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle podName:4cfb42b9-bb91-4012-879e-b185d1cce2eb nodeName:}" failed. No retries permitted until 2026-09-10 23:37:35.182745739 +0000 UTC m=+97.138977540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle") pod "router-default-84c49bb84b-8nk86" (UID: "4cfb42b9-bb91-4012-879e-b185d1cce2eb") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:19.184849 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.184819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cfb42b9-bb91-4012-879e-b185d1cce2eb-metrics-certs\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:19.209181 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.209159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"
Sep 10 23:37:19.300262 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.300231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.477127 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.477088    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7"]
Sep 10 23:37:19.691462 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.691426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.818578 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:19.818540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7" event={"ID":"74971d3e-ba66-4e20-8467-6aaba78f997d","Type":"ContainerStarted","Data":"5bae177e908d3833a50edc5a8ab1e5590c3aaae2327ac8174ea8589825f9904e"}
Sep 10 23:37:20.487372 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:20.487322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:20.597914 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:20.597879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:20.699182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:20.699139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:21.114467 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:21.114346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:21.295399 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:21.295364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:21.826138 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:21.826104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7" event={"ID":"74971d3e-ba66-4e20-8467-6aaba78f997d","Type":"ContainerStarted","Data":"a43697c8327e40d3161ec32b53a556dbde9f4d8f78586a21f35b3acd64f4bc4a"}
Sep 10 23:37:21.826512 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:21.826154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7" event={"ID":"74971d3e-ba66-4e20-8467-6aaba78f997d","Type":"ContainerStarted","Data":"793edcb26c2e525e60dfb5e71a79e62cd4ef3a8694b8acec54f63d34db4c18a0"}
Sep 10 23:37:21.843814 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:21.843765    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2jkv7" podStartSLOduration=17.225522983 podStartE2EDuration="18.843750632s" totalImagesPullingTime="1.618227649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:21.842811629 +0000 UTC m=+83.799043433" watchObservedRunningTime="2026-09-10 23:37:21.843750632 +0000 UTC m=+83.799982435"
Sep 10 23:37:21.881616 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:21.881582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:21.941553 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:21.941527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.288103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:22.288068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.691628 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:22.691540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.089333 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:23.089310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.486752 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:23.486667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.694323 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:23.694286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:24.093964 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:24.093929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:24.488789 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:24.488712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:24.892116 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:24.892078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:25.084088 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.084063    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:25.527950 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.527925    2578 scope.go:117] "RemoveContainer" containerID="2732e9784d8cade5661443ed477b2b0f6c7281dcd1ccf45c44618cfa8502e387"
Sep 10 23:37:25.840684 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.840613    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:37:25.841008 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.840993    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/1.log"
Sep 10 23:37:25.841063 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.841025    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a008749-a748-49c8-abc4-a0506f8f2e69" containerID="9339c416509f5bc0522b1df6898a9fe1f2dff00f17b5344fecb85c2785d29c79" exitCode=255
Sep 10 23:37:25.841099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.841067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" event={"ID":"2a008749-a748-49c8-abc4-a0506f8f2e69","Type":"ContainerDied","Data":"9339c416509f5bc0522b1df6898a9fe1f2dff00f17b5344fecb85c2785d29c79"}
Sep 10 23:37:25.841129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.841111    2578 scope.go:117] "RemoveContainer" containerID="2732e9784d8cade5661443ed477b2b0f6c7281dcd1ccf45c44618cfa8502e387"
Sep 10 23:37:25.841475 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:25.841455    2578 scope.go:117] "RemoveContainer" containerID="9339c416509f5bc0522b1df6898a9fe1f2dff00f17b5344fecb85c2785d29c79"
Sep 10 23:37:25.841638 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:25.841616    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-sq4f8_openshift-console-operator(2a008749-a748-49c8-abc4-a0506f8f2e69)\"" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" podUID="2a008749-a748-49c8-abc4-a0506f8f2e69"
Sep 10 23:37:26.643401 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:26.643372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:26.643780 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:26.643420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:26.643894 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:26.643874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:26.645677 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:26.645658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36fe9cad-01bc-41f0-8718-85cd9b5ba32d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrvpr\" (UID: \"36fe9cad-01bc-41f0-8718-85cd9b5ba32d\") " pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:26.796332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:26.796294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hrvpr"
Sep 10 23:37:26.850292 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:26.850170    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:37:27.070954 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:27.070932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hrvpr"]
Sep 10 23:37:27.073361 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:27.073334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fe9cad_01bc_41f0_8718_85cd9b5ba32d.slice/crio-eb0afff6081db1af1c3a03e1a113fd6f6d41ebc9f4acad4c9b03880212bb81fb WatchSource:0}: Error finding container eb0afff6081db1af1c3a03e1a113fd6f6d41ebc9f4acad4c9b03880212bb81fb: Status 404 returned error can't find the container with id eb0afff6081db1af1c3a03e1a113fd6f6d41ebc9f4acad4c9b03880212bb81fb
Sep 10 23:37:27.854791 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:27.854765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrvpr" event={"ID":"36fe9cad-01bc-41f0-8718-85cd9b5ba32d","Type":"ContainerStarted","Data":"ebe7dd055f1114a4428b2e06de2dde7fc3981fe9d3864fd29441f71dbf14616a"}
Sep 10 23:37:27.855055 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:27.854819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrvpr" event={"ID":"36fe9cad-01bc-41f0-8718-85cd9b5ba32d","Type":"ContainerStarted","Data":"eb0afff6081db1af1c3a03e1a113fd6f6d41ebc9f4acad4c9b03880212bb81fb"}
Sep 10 23:37:28.237429 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:28.237351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:28.338030 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:28.337996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:28.860981 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:28.860942    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrvpr" event={"ID":"36fe9cad-01bc-41f0-8718-85cd9b5ba32d","Type":"ContainerStarted","Data":"b90f6f905d165289f1507a5be2304404c7b516c52ab2e8ce9ee48c6dd2e5f4be"}
Sep 10 23:37:29.333147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:29.333117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:29.437653 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:29.437614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:29.536852 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:29.536812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:29.865940 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:29.865858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrvpr" event={"ID":"36fe9cad-01bc-41f0-8718-85cd9b5ba32d","Type":"ContainerStarted","Data":"a531fd93517014b861f19c3d6c045088e304e0da713f3af11c8c9d5f6447fad8"}
Sep 10 23:37:29.890254 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:29.890202    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hrvpr" podStartSLOduration=17.85407766 podStartE2EDuration="19.890186192s" totalImagesPullingTime="2.036108532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:29.889206841 +0000 UTC m=+91.845438641" watchObservedRunningTime="2026-09-10 23:37:29.890186192 +0000 UTC m=+91.846418003"
Sep 10 23:37:32.793740 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:32.793701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:32.854974 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:32.854937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:32.923599 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:32.923565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:33.525902 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:33.525861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:33.526078 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:33.525943    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:33.526229 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:33.526217    2578 scope.go:117] "RemoveContainer" containerID="9339c416509f5bc0522b1df6898a9fe1f2dff00f17b5344fecb85c2785d29c79"
Sep 10 23:37:33.526418 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:33.526398    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-sq4f8_openshift-console-operator(2a008749-a748-49c8-abc4-a0506f8f2e69)\"" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" podUID="2a008749-a748-49c8-abc4-a0506f8f2e69"
Sep 10 23:37:35.204611 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:35.204572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:35.205197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:35.205175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfb42b9-bb91-4012-879e-b185d1cce2eb-service-ca-bundle\") pod \"router-default-84c49bb84b-8nk86\" (UID: \"4cfb42b9-bb91-4012-879e-b185d1cce2eb\") " pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:35.219641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:35.219606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:35.511226 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:35.511200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84c49bb84b-8nk86"]
Sep 10 23:37:35.512929 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:35.512901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfb42b9_bb91_4012_879e_b185d1cce2eb.slice/crio-76e8d0a6968332a4e862e3695c305dd2194b3a498e6404d66350dfd45819584f WatchSource:0}: Error finding container 76e8d0a6968332a4e862e3695c305dd2194b3a498e6404d66350dfd45819584f: Status 404 returned error can't find the container with id 76e8d0a6968332a4e862e3695c305dd2194b3a498e6404d66350dfd45819584f
Sep 10 23:37:35.886550 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:35.886514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84c49bb84b-8nk86" event={"ID":"4cfb42b9-bb91-4012-879e-b185d1cce2eb","Type":"ContainerStarted","Data":"5551904323fe5d44f31ad020b2ab3276567365364e89e90c6ca50e5b7c6019bd"}
Sep 10 23:37:35.886727 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:35.886564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84c49bb84b-8nk86" event={"ID":"4cfb42b9-bb91-4012-879e-b185d1cce2eb","Type":"ContainerStarted","Data":"76e8d0a6968332a4e862e3695c305dd2194b3a498e6404d66350dfd45819584f"}
Sep 10 23:37:35.913087 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:35.913039    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-84c49bb84b-8nk86" podStartSLOduration=32.913022785 podStartE2EDuration="32.913022785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:35.912330257 +0000 UTC m=+97.868562059" watchObservedRunningTime="2026-09-10 23:37:35.913022785 +0000 UTC m=+97.869254588"
Sep 10 23:37:36.219849 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.219765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:36.222184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.222163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:36.316003 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.315969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:37:36.316155 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.316055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:37:36.318327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.318295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1c2427c-a1e9-4e53-a2de-e7b8630507fe-metrics-tls\") pod \"dns-default-bzgp5\" (UID: \"f1c2427c-a1e9-4e53-a2de-e7b8630507fe\") " pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:37:36.318495 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.318474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"image-registry-6b7f6bf58d-tvcbw\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") " pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:37:36.353360 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.353329    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vhrn5\""
Sep 10 23:37:36.360863 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.360840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:37:36.366395 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.366379    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9kf9v\""
Sep 10 23:37:36.373784 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.373763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:37:36.638346 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.638311    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"]
Sep 10 23:37:36.641640 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:36.641614    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4caf2f_2748_4852_afc6_19f7bf42c34e.slice/crio-df68e47f75c82807193607d0e3ee3fa137aa1aca98fe2adabc72c7b438744fd8 WatchSource:0}: Error finding container df68e47f75c82807193607d0e3ee3fa137aa1aca98fe2adabc72c7b438744fd8: Status 404 returned error can't find the container with id df68e47f75c82807193607d0e3ee3fa137aa1aca98fe2adabc72c7b438744fd8
Sep 10 23:37:36.650378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.650353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bzgp5"]
Sep 10 23:37:36.653532 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:36.653508    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c2427c_a1e9_4e53_a2de_e7b8630507fe.slice/crio-a08b32f53f1c69ccfc0f0c050f421728c037159115b6cef73731cb95bbd91cb4 WatchSource:0}: Error finding container a08b32f53f1c69ccfc0f0c050f421728c037159115b6cef73731cb95bbd91cb4: Status 404 returned error can't find the container with id a08b32f53f1c69ccfc0f0c050f421728c037159115b6cef73731cb95bbd91cb4
Sep 10 23:37:36.890991 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.890905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bzgp5" event={"ID":"f1c2427c-a1e9-4e53-a2de-e7b8630507fe","Type":"ContainerStarted","Data":"a08b32f53f1c69ccfc0f0c050f421728c037159115b6cef73731cb95bbd91cb4"}
Sep 10 23:37:36.892645 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.892614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw" event={"ID":"2f4caf2f-2748-4852-afc6-19f7bf42c34e","Type":"ContainerStarted","Data":"d44d0d61a9d7b8493ba26731107f6bc03f935ae888c0130137ae418ac1e21085"}
Sep 10 23:37:36.892781 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.892664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw" event={"ID":"2f4caf2f-2748-4852-afc6-19f7bf42c34e","Type":"ContainerStarted","Data":"df68e47f75c82807193607d0e3ee3fa137aa1aca98fe2adabc72c7b438744fd8"}
Sep 10 23:37:36.893100 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.893076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:36.894232 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.894210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84c49bb84b-8nk86"
Sep 10 23:37:36.926774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:36.926714    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw" podStartSLOduration=96.926691632 podStartE2EDuration="1m36.926691632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:36.924700476 +0000 UTC m=+98.880932279" watchObservedRunningTime="2026-09-10 23:37:36.926691632 +0000 UTC m=+98.882923436"
Sep 10 23:37:37.103660 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:37.103628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:37.193859 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:37.193774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:37.896526 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:37.896487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:37:38.047336 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.047249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.149221 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.149159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.428142 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.428060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.529733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.529701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.602049 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.602015    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"]
Sep 10 23:37:38.605002 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.604986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"
Sep 10 23:37:38.607744 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.607717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 23:37:38.607840 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.607819    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8d99j\""
Sep 10 23:37:38.614015 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.613968    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"]
Sep 10 23:37:38.614509 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.614483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.737192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.737104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5da565e0-536c-416a-9366-98d8ce4e62db-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qsf5b\" (UID: \"5da565e0-536c-416a-9366-98d8ce4e62db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"
Sep 10 23:37:38.805793 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.805760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.838305 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.838256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5da565e0-536c-416a-9366-98d8ce4e62db-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qsf5b\" (UID: \"5da565e0-536c-416a-9366-98d8ce4e62db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"
Sep 10 23:37:38.840615 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.840587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5da565e0-536c-416a-9366-98d8ce4e62db-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qsf5b\" (UID: \"5da565e0-536c-416a-9366-98d8ce4e62db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"
Sep 10 23:37:38.900702 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.900670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bzgp5" event={"ID":"f1c2427c-a1e9-4e53-a2de-e7b8630507fe","Type":"ContainerStarted","Data":"b08e11405afd8199b568e17af296c9355c733ce64c6468e73396e195a37841c7"}
Sep 10 23:37:38.901077 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.900718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bzgp5" event={"ID":"f1c2427c-a1e9-4e53-a2de-e7b8630507fe","Type":"ContainerStarted","Data":"2d699a75c5c0c50f8bbe122500ee95085a845cf2ea834d6d4f7a2ecb6df414db"}
Sep 10 23:37:38.901077 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.900820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:37:38.913902 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.913880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"
Sep 10 23:37:38.918338 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:38.918260    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bzgp5" podStartSLOduration=65.476782948 podStartE2EDuration="1m6.918241335s" totalImagesPullingTime="1.441458387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:38.917202166 +0000 UTC m=+100.873433968" watchObservedRunningTime="2026-09-10 23:37:38.918241335 +0000 UTC m=+100.874473139"
Sep 10 23:37:39.179146 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:39.178866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"]
Sep 10 23:37:39.181568 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:39.181537    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da565e0_536c_416a_9366_98d8ce4e62db.slice/crio-8df2fb1361ac3021b1233787ec5983c825ed58b2e1179978bec04071bec15cfb WatchSource:0}: Error finding container 8df2fb1361ac3021b1233787ec5983c825ed58b2e1179978bec04071bec15cfb: Status 404 returned error can't find the container with id 8df2fb1361ac3021b1233787ec5983c825ed58b2e1179978bec04071bec15cfb
Sep 10 23:37:39.906747 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:39.906711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b" event={"ID":"5da565e0-536c-416a-9366-98d8ce4e62db","Type":"ContainerStarted","Data":"8df2fb1361ac3021b1233787ec5983c825ed58b2e1179978bec04071bec15cfb"}
Sep 10 23:37:40.911322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:40.911268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b" event={"ID":"5da565e0-536c-416a-9366-98d8ce4e62db","Type":"ContainerStarted","Data":"b73e31fc7f2e87d1e0245c4267bfc79d88f1fcb4304a0927715ee855a64b60bb"}
Sep 10 23:37:40.911783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:40.911432    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"
Sep 10 23:37:40.916354 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:40.916332    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b"
Sep 10 23:37:40.927148 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:40.927103    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qsf5b" podStartSLOduration=1.386039842 podStartE2EDuration="2.927089428s" totalImagesPullingTime="1.541049586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:40.925924826 +0000 UTC m=+102.882156639" watchObservedRunningTime="2026-09-10 23:37:40.927089428 +0000 UTC m=+102.883321276"
Sep 10 23:37:41.613103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.613073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:41.650244 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.650218    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"]
Sep 10 23:37:41.664792 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.664768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"]
Sep 10 23:37:41.664888 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.664840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.667856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.667836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 23:37:41.669033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.668820    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 23:37:41.669033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.668834    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 23:37:41.669033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.668825    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 23:37:41.669033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.668825    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 23:37:41.669033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.668984    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-p6h7r\""
Sep 10 23:37:41.684193 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.684170    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:41.759753 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.759720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac4929b-70d2-4596-a915-a74186b90320-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.759852 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.759761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8fw\" (UniqueName: \"kubernetes.io/projected/1ac4929b-70d2-4596-a915-a74186b90320-kube-api-access-zl8fw\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.759852 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.759840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ac4929b-70d2-4596-a915-a74186b90320-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.759921 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.759865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac4929b-70d2-4596-a915-a74186b90320-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.799136 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.799117    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vk9qm"
Sep 10 23:37:41.840724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.840686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:41.860457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.860423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ac4929b-70d2-4596-a915-a74186b90320-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.860609 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.860472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac4929b-70d2-4596-a915-a74186b90320-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.860609 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.860537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac4929b-70d2-4596-a915-a74186b90320-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.860609 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.860575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8fw\" (UniqueName: \"kubernetes.io/projected/1ac4929b-70d2-4596-a915-a74186b90320-kube-api-access-zl8fw\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.861233 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.861205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac4929b-70d2-4596-a915-a74186b90320-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.862679 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.862649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ac4929b-70d2-4596-a915-a74186b90320-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.863051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.863030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ac4929b-70d2-4596-a915-a74186b90320-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.869260 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.869206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8fw\" (UniqueName: \"kubernetes.io/projected/1ac4929b-70d2-4596-a915-a74186b90320-kube-api-access-zl8fw\") pod \"prometheus-operator-68bcd7f987-wnhww\" (UID: \"1ac4929b-70d2-4596-a915-a74186b90320\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:41.943378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.943346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:41.974347 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:41.974315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"
Sep 10 23:37:42.046076 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:42.046046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:42.110096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:42.110067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:42.282335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:42.282308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wnhww"]
Sep 10 23:37:42.285515 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:42.285493    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac4929b_70d2_4596_a915_a74186b90320.slice/crio-4e2878ef8a71c02036fec83ef24c6aa0fe42744fd0251a58b1e8686eca2b2e7b WatchSource:0}: Error finding container 4e2878ef8a71c02036fec83ef24c6aa0fe42744fd0251a58b1e8686eca2b2e7b: Status 404 returned error can't find the container with id 4e2878ef8a71c02036fec83ef24c6aa0fe42744fd0251a58b1e8686eca2b2e7b
Sep 10 23:37:42.920312 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:42.920263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww" event={"ID":"1ac4929b-70d2-4596-a915-a74186b90320","Type":"ContainerStarted","Data":"4e2878ef8a71c02036fec83ef24c6aa0fe42744fd0251a58b1e8686eca2b2e7b"}
Sep 10 23:37:43.240473 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:43.240391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:43.344704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:43.344672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:44.177982 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:44.177909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:44.537094 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:44.537059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:44.638257 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:44.638220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:44.928137 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:44.928063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww" event={"ID":"1ac4929b-70d2-4596-a915-a74186b90320","Type":"ContainerStarted","Data":"744af17b89b5a8183d684fb83330a46040a937a1c230212ec4e48c97348bc78f"}
Sep 10 23:37:44.928137 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:44.928112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww" event={"ID":"1ac4929b-70d2-4596-a915-a74186b90320","Type":"ContainerStarted","Data":"e4a62c31b5142a044d125429d3c7797b63282f7ed7510879ebe6bd1b6f9c77f8"}
Sep 10 23:37:44.946138 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:44.946094    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wnhww" podStartSLOduration=2.332326749 podStartE2EDuration="3.946078999s" totalImagesPullingTime="1.61375225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:44.944110875 +0000 UTC m=+106.900342682" watchObservedRunningTime="2026-09-10 23:37:44.946078999 +0000 UTC m=+106.902310802"
Sep 10 23:37:45.895799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:45.895756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:45.948597 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:45.948557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:46.528628 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:46.528597    2578 scope.go:117] "RemoveContainer" containerID="9339c416509f5bc0522b1df6898a9fe1f2dff00f17b5344fecb85c2785d29c79"
Sep 10 23:37:46.794197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:46.794116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:46.937769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:46.937745    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:37:46.938230 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:46.937792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" event={"ID":"2a008749-a748-49c8-abc4-a0506f8f2e69","Type":"ContainerStarted","Data":"16b7df99a5cfe347b10687d0eb42a219bdc94c21151b6455a033be1afb5621cf"}
Sep 10 23:37:46.938230 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:46.938055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:46.956546 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:46.956501    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8" podStartSLOduration=41.298300157 podStartE2EDuration="43.956488074s" totalImagesPullingTime="2.658187917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:46.955598878 +0000 UTC m=+108.911830694" watchObservedRunningTime="2026-09-10 23:37:46.956488074 +0000 UTC m=+108.912719877"
Sep 10 23:37:46.987353 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:46.987321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.008671 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.008645    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"]
Sep 10 23:37:47.011576 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.011558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.011845 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.011826    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"]
Sep 10 23:37:47.014396 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.014377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.014845 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.014557    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dmpf2\""
Sep 10 23:37:47.014845 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.014604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 23:37:47.016425 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.016411    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 23:37:47.016760 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.016742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 23:37:47.017017 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.017002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 23:37:47.017719 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.017700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b476q\""
Sep 10 23:37:47.017823 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.017703    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 23:37:47.023967 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.023949    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fpb6j"]
Sep 10 23:37:47.028255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.028237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"]
Sep 10 23:37:47.028355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.028291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"]
Sep 10 23:37:47.028419 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.028381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.034120 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.034103    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gv69c\""
Sep 10 23:37:47.034230 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.034172    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 23:37:47.034395 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.034377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 23:37:47.034470 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.034419    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 23:37:47.084178 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.084086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.095132 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.095132 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.095340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.095340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.095340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.095340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886s6\" (UniqueName: \"kubernetes.io/projected/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-kube-api-access-886s6\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.095340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbq4\" (UniqueName: \"kubernetes.io/projected/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-api-access-tvbq4\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.095534 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.095534 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.095534 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.095470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.180963 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.180925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.196010 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.195990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-root\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196081 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.196081 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-textfile\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196081 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.196182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.196182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886s6\" (UniqueName: \"kubernetes.io/projected/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-kube-api-access-886s6\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.196254 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbq4\" (UniqueName: \"kubernetes.io/projected/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-api-access-tvbq4\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.196336 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196336 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.196441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.196441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-wtmp\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-sys\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196624 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:47.196506    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 10 23:37:47.196624 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:37:47.196600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-tls podName:cb87bdeb-36f6-4a0e-a3a8-e5f436232321 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:47.696580069 +0000 UTC m=+109.652811867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-j9fhz" (UID: "cb87bdeb-36f6-4a0e-a3a8-e5f436232321") : secret "kube-state-metrics-tls" not found
Sep 10 23:37:47.196746 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.196798 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-metrics-client-ca\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196850 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.196850 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cnt\" (UniqueName: \"kubernetes.io/projected/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-kube-api-access-p9cnt\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.196959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.196959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.197057 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.196958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-tls\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.197184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.197143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.197304 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.197267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.197489 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.197465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.198702 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.198674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.198783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.198700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.199247 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.199226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.208701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.208673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbq4\" (UniqueName: \"kubernetes.io/projected/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-api-access-tvbq4\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.208777 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.208732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-886s6\" (UniqueName: \"kubernetes.io/projected/1faa6ed1-d13b-4b81-aeaf-125eb3ccc358-kube-api-access-886s6\") pod \"openshift-state-metrics-67dcf8c979-dl42r\" (UID: \"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.298091 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-metrics-client-ca\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cnt\" (UniqueName: \"kubernetes.io/projected/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-kube-api-access-p9cnt\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-tls\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-root\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298408 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-textfile\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298408 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-root\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298408 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-wtmp\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-sys\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-textfile\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-sys\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-wtmp\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-metrics-client-ca\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.298994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.298973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.300385 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.300365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.300799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.300776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-node-exporter-tls\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.306185 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.306164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cnt\" (UniqueName: \"kubernetes.io/projected/3be8851d-6eba-4ac5-9ba1-13a13caeec1d-kube-api-access-p9cnt\") pod \"node-exporter-fpb6j\" (UID: \"3be8851d-6eba-4ac5-9ba1-13a13caeec1d\") " pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.323115 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.323093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"
Sep 10 23:37:47.339083 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.339060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fpb6j"
Sep 10 23:37:47.350101 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:47.350073    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be8851d_6eba_4ac5_9ba1_13a13caeec1d.slice/crio-8297115d429ca9e68d5497da4a9fd62cc92a491571780ee9c079f5f5cdabc097 WatchSource:0}: Error finding container 8297115d429ca9e68d5497da4a9fd62cc92a491571780ee9c079f5f5cdabc097: Status 404 returned error can't find the container with id 8297115d429ca9e68d5497da4a9fd62cc92a491571780ee9c079f5f5cdabc097
Sep 10 23:37:47.351059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.351031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.436457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.436424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-sq4f8"
Sep 10 23:37:47.452147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.452058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.582992 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.582482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.610321 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.609959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r"]
Sep 10 23:37:47.613547 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:47.613515    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faa6ed1_d13b_4b81_aeaf_125eb3ccc358.slice/crio-2eae4f63993aae5f7832e1e3faf8be0da0a60cb5763c04a8a01e865a47ca7760 WatchSource:0}: Error finding container 2eae4f63993aae5f7832e1e3faf8be0da0a60cb5763c04a8a01e865a47ca7760: Status 404 returned error can't find the container with id 2eae4f63993aae5f7832e1e3faf8be0da0a60cb5763c04a8a01e865a47ca7760
Sep 10 23:37:47.623616 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.623591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-jnpnm"]
Sep 10 23:37:47.626832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.626815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-jnpnm"
Sep 10 23:37:47.629521 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.629498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fx8db\""
Sep 10 23:37:47.629638 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.629618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 23:37:47.629686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.629637    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 23:37:47.638295 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.638258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-jnpnm"]
Sep 10 23:37:47.674169 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.674138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.702810 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.702786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.705181 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.705159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb87bdeb-36f6-4a0e-a3a8-e5f436232321-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-j9fhz\" (UID: \"cb87bdeb-36f6-4a0e-a3a8-e5f436232321\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.775024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.774990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.804165 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.804134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6xt\" (UniqueName: \"kubernetes.io/projected/25f6aa54-56b3-454a-81b6-fe8a56a31fb3-kube-api-access-lm6xt\") pod \"downloads-76f99584f4-jnpnm\" (UID: \"25f6aa54-56b3-454a-81b6-fe8a56a31fb3\") " pod="openshift-console/downloads-76f99584f4-jnpnm"
Sep 10 23:37:47.905378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.905293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6xt\" (UniqueName: \"kubernetes.io/projected/25f6aa54-56b3-454a-81b6-fe8a56a31fb3-kube-api-access-lm6xt\") pod \"downloads-76f99584f4-jnpnm\" (UID: \"25f6aa54-56b3-454a-81b6-fe8a56a31fb3\") " pod="openshift-console/downloads-76f99584f4-jnpnm"
Sep 10 23:37:47.914155 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.914118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6xt\" (UniqueName: \"kubernetes.io/projected/25f6aa54-56b3-454a-81b6-fe8a56a31fb3-kube-api-access-lm6xt\") pod \"downloads-76f99584f4-jnpnm\" (UID: \"25f6aa54-56b3-454a-81b6-fe8a56a31fb3\") " pod="openshift-console/downloads-76f99584f4-jnpnm"
Sep 10 23:37:47.929912 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.929887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"
Sep 10 23:37:47.939619 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.939551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.943783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.943758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r" event={"ID":"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358","Type":"ContainerStarted","Data":"077bbc727c4c6ae12f4653beda8587c170d3dd897d9d27ddd6b0a63b9d751959"}
Sep 10 23:37:47.943890 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.943820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r" event={"ID":"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358","Type":"ContainerStarted","Data":"51c23ace2eb9f1caf22a86f5c8348d6170a0d31be4f35a3a544e489e2b77b5c4"}
Sep 10 23:37:47.943890 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.943852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r" event={"ID":"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358","Type":"ContainerStarted","Data":"2eae4f63993aae5f7832e1e3faf8be0da0a60cb5763c04a8a01e865a47ca7760"}
Sep 10 23:37:47.945332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.945306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpb6j" event={"ID":"3be8851d-6eba-4ac5-9ba1-13a13caeec1d","Type":"ContainerStarted","Data":"8297115d429ca9e68d5497da4a9fd62cc92a491571780ee9c079f5f5cdabc097"}
Sep 10 23:37:47.955693 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:47.955668    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-jnpnm"
Sep 10 23:37:48.040048 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.039960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.138397 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.138363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.224461 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.224414    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz"]
Sep 10 23:37:48.254150 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.254032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-jnpnm"]
Sep 10 23:37:48.354403 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.354335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.392609 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:48.392574    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb87bdeb_36f6_4a0e_a3a8_e5f436232321.slice/crio-b39bb174d3b6f2f5ab76f1330e75ed3d13a8e96112eaf709d2bf2047784330b8 WatchSource:0}: Error finding container b39bb174d3b6f2f5ab76f1330e75ed3d13a8e96112eaf709d2bf2047784330b8: Status 404 returned error can't find the container with id b39bb174d3b6f2f5ab76f1330e75ed3d13a8e96112eaf709d2bf2047784330b8
Sep 10 23:37:48.393654 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:48.393628    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f6aa54_56b3_454a_81b6_fe8a56a31fb3.slice/crio-8fb87ffdf6cb133f0754116000a86c2232c1e9f1d95831f5b4988ee4d2117ee8 WatchSource:0}: Error finding container 8fb87ffdf6cb133f0754116000a86c2232c1e9f1d95831f5b4988ee4d2117ee8: Status 404 returned error can't find the container with id 8fb87ffdf6cb133f0754116000a86c2232c1e9f1d95831f5b4988ee4d2117ee8
Sep 10 23:37:48.478642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.478590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.723088 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.723024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.840772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.840749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.908881 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.908850    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bzgp5"
Sep 10 23:37:48.954856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.954736    2578 generic.go:356] "Generic (PLEG): container finished" podID="3be8851d-6eba-4ac5-9ba1-13a13caeec1d" containerID="4f4e777a62d49c8a3b88da494280bd31b7ffe916bc0328788199ec7022bc22f9" exitCode=0
Sep 10 23:37:48.954856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.954773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpb6j" event={"ID":"3be8851d-6eba-4ac5-9ba1-13a13caeec1d","Type":"ContainerDied","Data":"4f4e777a62d49c8a3b88da494280bd31b7ffe916bc0328788199ec7022bc22f9"}
Sep 10 23:37:48.957354 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.957263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz" event={"ID":"cb87bdeb-36f6-4a0e-a3a8-e5f436232321","Type":"ContainerStarted","Data":"b39bb174d3b6f2f5ab76f1330e75ed3d13a8e96112eaf709d2bf2047784330b8"}
Sep 10 23:37:48.962239 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.962212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r" event={"ID":"1faa6ed1-d13b-4b81-aeaf-125eb3ccc358","Type":"ContainerStarted","Data":"7e007c0e9d97d9d9d4e15f6a649b028ef8705542aaf0634a028ae54d505bd9dc"}
Sep 10 23:37:48.966583 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.966533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-jnpnm" event={"ID":"25f6aa54-56b3-454a-81b6-fe8a56a31fb3","Type":"ContainerStarted","Data":"8fb87ffdf6cb133f0754116000a86c2232c1e9f1d95831f5b4988ee4d2117ee8"}
Sep 10 23:37:48.969457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:48.969427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.018149 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.018098    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dl42r" podStartSLOduration=1.960317357 podStartE2EDuration="3.018081715s" totalImagesPullingTime="1.057764358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:49.016752154 +0000 UTC m=+110.972983953" watchObservedRunningTime="2026-09-10 23:37:49.018081715 +0000 UTC m=+110.974313528"
Sep 10 23:37:49.248055 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.247941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.457254 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.457142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.556544 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.556503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.937822 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.937791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.975304 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.975229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpb6j" event={"ID":"3be8851d-6eba-4ac5-9ba1-13a13caeec1d","Type":"ContainerStarted","Data":"42ff6a8876712eadf2f244a4a523bcd9132c23c4cf8800f6da6c5e9f24004ea7"}
Sep 10 23:37:49.975912 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.975338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpb6j" event={"ID":"3be8851d-6eba-4ac5-9ba1-13a13caeec1d","Type":"ContainerStarted","Data":"8c6eb184911e92ff37512d0111f88fde1ba84e3191a53a8ab10870f8352d7603"}
Sep 10 23:37:49.978781 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.978751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz" event={"ID":"cb87bdeb-36f6-4a0e-a3a8-e5f436232321","Type":"ContainerStarted","Data":"e7ca8de303727dd5048e8a32d54d67b85676f492de2e1316c13aa3290143f420"}
Sep 10 23:37:49.978873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.978814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz" event={"ID":"cb87bdeb-36f6-4a0e-a3a8-e5f436232321","Type":"ContainerStarted","Data":"794552c9be3f44c0648a3e4da824a5b9321e69e9c9c2f11ab41c16501e94e22d"}
Sep 10 23:37:49.978873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.978843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz" event={"ID":"cb87bdeb-36f6-4a0e-a3a8-e5f436232321","Type":"ContainerStarted","Data":"5e922f37d36a6766cfde078aa5f7e404e9c66cdb3abd6915b3cbe6805537ba36"}
Sep 10 23:37:49.998496 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:49.998405    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fpb6j" podStartSLOduration=1.838244097 podStartE2EDuration="2.99838358s" totalImagesPullingTime="1.08749153s" totalInitContainerRuntime="72.647953ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:49.998136739 +0000 UTC m=+111.954368542" watchObservedRunningTime="2026-09-10 23:37:49.99838358 +0000 UTC m=+111.954615373"
Sep 10 23:37:50.003624 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:50.003596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:50.024291 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:50.024230    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-j9fhz" podStartSLOduration=2.676040742 podStartE2EDuration="4.024211426s" totalImagesPullingTime="1.348170684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:50.022757228 +0000 UTC m=+111.978989031" watchObservedRunningTime="2026-09-10 23:37:50.024211426 +0000 UTC m=+111.980443231"
Sep 10 23:37:50.130551 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:50.130516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:50.222292 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:50.222233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:50.321750 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:50.321703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:50.423344 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:50.423253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.448030 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.447992    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78846476b6-cbm59"]
Sep 10 23:37:51.454995 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.454974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.459500 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.459475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fvchw\""
Sep 10 23:37:51.460757 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.460671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 23:37:51.460757 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.460683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 23:37:51.461022 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.460838    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eur14i964maqs\""
Sep 10 23:37:51.461022 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.460899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 23:37:51.463689 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.463671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 23:37:51.470222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.470196    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78846476b6-cbm59"]
Sep 10 23:37:51.646642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.646613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-secret-metrics-server-tls\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.646799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.646654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79088aca-c73e-452a-9002-2e953505018a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.646799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.646698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzd9\" (UniqueName: \"kubernetes.io/projected/79088aca-c73e-452a-9002-2e953505018a-kube-api-access-phzd9\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.646799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.646782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-secret-metrics-server-client-certs\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.646956 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.646820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/79088aca-c73e-452a-9002-2e953505018a-audit-log\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.646956 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.646845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-client-ca-bundle\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.646956 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.646939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/79088aca-c73e-452a-9002-2e953505018a-metrics-server-audit-profiles\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.748400 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.748258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzd9\" (UniqueName: \"kubernetes.io/projected/79088aca-c73e-452a-9002-2e953505018a-kube-api-access-phzd9\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.748559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.748459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-secret-metrics-server-client-certs\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.748559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.748527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/79088aca-c73e-452a-9002-2e953505018a-audit-log\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.748559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.748555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-client-ca-bundle\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.748687 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.748598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/79088aca-c73e-452a-9002-2e953505018a-metrics-server-audit-profiles\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.748687 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.748653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-secret-metrics-server-tls\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.748751 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.748687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79088aca-c73e-452a-9002-2e953505018a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.749314 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.749012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/79088aca-c73e-452a-9002-2e953505018a-audit-log\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.749580 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.749534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79088aca-c73e-452a-9002-2e953505018a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.749975 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.749949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/79088aca-c73e-452a-9002-2e953505018a-metrics-server-audit-profiles\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.751906 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.751845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-client-ca-bundle\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.753733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.753707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-secret-metrics-server-tls\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.754024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.754002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/79088aca-c73e-452a-9002-2e953505018a-secret-metrics-server-client-certs\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.760083 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.759555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzd9\" (UniqueName: \"kubernetes.io/projected/79088aca-c73e-452a-9002-2e953505018a-kube-api-access-phzd9\") pod \"metrics-server-78846476b6-cbm59\" (UID: \"79088aca-c73e-452a-9002-2e953505018a\") " pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.766293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.766259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:37:51.789328 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.789259    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.797834 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.797803    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"]
Sep 10 23:37:51.803040 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.803020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:51.805853 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.805835    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 23:37:51.805943 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.805871    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4mt5k\""
Sep 10 23:37:51.811642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.811617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"]
Sep 10 23:37:51.888118 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.888081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.951288 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.951239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a01c458-2e83-4321-9ba8-5da89b87ef3c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5pf98\" (UID: \"3a01c458-2e83-4321-9ba8-5da89b87ef3c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:51.951433 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:51.951311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qwk\" (UniqueName: \"kubernetes.io/projected/3a01c458-2e83-4321-9ba8-5da89b87ef3c-kube-api-access-r8qwk\") pod \"monitoring-plugin-5d85778b9d-5pf98\" (UID: \"3a01c458-2e83-4321-9ba8-5da89b87ef3c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:52.013506 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.013437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:52.052455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.052419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a01c458-2e83-4321-9ba8-5da89b87ef3c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5pf98\" (UID: \"3a01c458-2e83-4321-9ba8-5da89b87ef3c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:52.052455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.052456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qwk\" (UniqueName: \"kubernetes.io/projected/3a01c458-2e83-4321-9ba8-5da89b87ef3c-kube-api-access-r8qwk\") pod \"monitoring-plugin-5d85778b9d-5pf98\" (UID: \"3a01c458-2e83-4321-9ba8-5da89b87ef3c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:52.055584 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.055563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a01c458-2e83-4321-9ba8-5da89b87ef3c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5pf98\" (UID: \"3a01c458-2e83-4321-9ba8-5da89b87ef3c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:52.055874 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.055854    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78846476b6-cbm59"]
Sep 10 23:37:52.058030 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:52.057998    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79088aca_c73e_452a_9002_2e953505018a.slice/crio-6d607b713e03283c879f6c5719eb59bc06c8d1fa9ff655e86e3364dc94f8885b WatchSource:0}: Error finding container 6d607b713e03283c879f6c5719eb59bc06c8d1fa9ff655e86e3364dc94f8885b: Status 404 returned error can't find the container with id 6d607b713e03283c879f6c5719eb59bc06c8d1fa9ff655e86e3364dc94f8885b
Sep 10 23:37:52.061645 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.061621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qwk\" (UniqueName: \"kubernetes.io/projected/3a01c458-2e83-4321-9ba8-5da89b87ef3c-kube-api-access-r8qwk\") pod \"monitoring-plugin-5d85778b9d-5pf98\" (UID: \"3a01c458-2e83-4321-9ba8-5da89b87ef3c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:52.115414 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.115385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:52.391188 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.391108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"]
Sep 10 23:37:52.394195 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:52.394163    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a01c458_2e83_4321_9ba8_5da89b87ef3c.slice/crio-9600c0b7fcd4756869af054f72abb0a9b4600ca6b80effe18e1f3cd737901b9c WatchSource:0}: Error finding container 9600c0b7fcd4756869af054f72abb0a9b4600ca6b80effe18e1f3cd737901b9c: Status 404 returned error can't find the container with id 9600c0b7fcd4756869af054f72abb0a9b4600ca6b80effe18e1f3cd737901b9c
Sep 10 23:37:52.940870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.940316    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"]
Sep 10 23:37:52.946502 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.946472    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:37:52.995136 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.995098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98" event={"ID":"3a01c458-2e83-4321-9ba8-5da89b87ef3c","Type":"ContainerStarted","Data":"9600c0b7fcd4756869af054f72abb0a9b4600ca6b80effe18e1f3cd737901b9c"}
Sep 10 23:37:52.996959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:52.996933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78846476b6-cbm59" event={"ID":"79088aca-c73e-452a-9002-2e953505018a","Type":"ContainerStarted","Data":"6d607b713e03283c879f6c5719eb59bc06c8d1fa9ff655e86e3364dc94f8885b"}
Sep 10 23:37:53.063091 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.063032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.173032 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.172980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.262601 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.262157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.529702 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.529292    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:37:53.535873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.535845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.540149 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.540128    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 23:37:53.540480 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.540357    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 23:37:53.540649 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.540607    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 23:37:53.540649 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.540619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 23:37:53.543197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.541331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 23:37:53.543197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.541561    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 23:37:53.543197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.542041    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 23:37:53.543197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.542225    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 23:37:53.543197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.542410    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 23:37:53.543197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.542567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 23:37:53.545156 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.545136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 23:37:53.545295 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.545200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a081n4fej99kg\""
Sep 10 23:37:53.546250 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.546078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 23:37:53.546434 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.546360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.548457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.548437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kwjln\""
Sep 10 23:37:53.551165 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.550943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 23:37:53.556933 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.556911    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 23:37:53.567346 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.567319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:37:53.569346 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.569256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.569534 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.569516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.569694 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.569673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.569853 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.569820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570034 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570167 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570311 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570388 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrjl\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-kube-api-access-vtrjl\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.570772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.570711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.645667 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.645584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.671205 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671205 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrjl\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-kube-api-access-vtrjl\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.671820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.671792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.672599 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.672542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.672949 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.672924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.674226 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.673906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.674738 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.674710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.675146 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.674819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.678910 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.678885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.679224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.679203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.680995 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.680966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.682335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.682226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.683253 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.682776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.683253 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.683216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.683579 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.683457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.684464 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.684429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.684876 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.684822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.685786 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.685761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.685786 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.685776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.686180 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.686161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.686635 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.686614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.688767 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.688748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.692901 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.692873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrjl\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-kube-api-access-vtrjl\") pod \"prometheus-k8s-0\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:53.742751 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.742709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.852733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:53.852402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:54.624106 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:54.624076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:37:54.626081 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:37:54.626038    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbb1027_4e0b_496c_bca7_e2af674f39d9.slice/crio-404bd2051162fd5007f1910a87c13afffb00b0465ac9d0cac9a59f85673591fe WatchSource:0}: Error finding container 404bd2051162fd5007f1910a87c13afffb00b0465ac9d0cac9a59f85673591fe: Status 404 returned error can't find the container with id 404bd2051162fd5007f1910a87c13afffb00b0465ac9d0cac9a59f85673591fe
Sep 10 23:37:55.008407 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:55.008298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerStarted","Data":"404bd2051162fd5007f1910a87c13afffb00b0465ac9d0cac9a59f85673591fe"}
Sep 10 23:37:55.010019 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:55.009988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98" event={"ID":"3a01c458-2e83-4321-9ba8-5da89b87ef3c","Type":"ContainerStarted","Data":"93be73978baba458d57c6a65d3f4fdb99c483045701da00a02a9f626227277de"}
Sep 10 23:37:55.010220 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:55.010195    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:55.012295 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:55.012242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78846476b6-cbm59" event={"ID":"79088aca-c73e-452a-9002-2e953505018a","Type":"ContainerStarted","Data":"f5ec591c2d59ab212da601d178fa689bb9a74c7a188d95d63b2721221ead50de"}
Sep 10 23:37:55.016210 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:55.016164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98"
Sep 10 23:37:55.026587 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:55.026450    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5pf98" podStartSLOduration=2.08143302 podStartE2EDuration="4.026433805s" totalImagesPullingTime="1.945000785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:55.025942791 +0000 UTC m=+116.982174599" watchObservedRunningTime="2026-09-10 23:37:55.026433805 +0000 UTC m=+116.982665609"
Sep 10 23:37:55.045181 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:55.044476    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78846476b6-cbm59" podStartSLOduration=1.7654449140000001 podStartE2EDuration="4.044456325s" totalImagesPullingTime="2.279011411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:55.042876371 +0000 UTC m=+116.999108176" watchObservedRunningTime="2026-09-10 23:37:55.044456325 +0000 UTC m=+117.000688129"
Sep 10 23:37:56.019015 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:56.018979    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerID="392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3" exitCode=0
Sep 10 23:37:56.019473 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:56.019069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3"}
Sep 10 23:37:56.039705 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:56.039676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:56.140927 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:56.140889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:58.922507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:37:58.922467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:01.326675 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:01.326617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:01.427432 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:01.427393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:04.737491 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:04.737434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:04.797598 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:04.797556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:05.046371 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:05.046338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:05.103541 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:05.103492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:06.130332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:06.130267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:06.230107 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:06.230071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:07.071960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.071916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-jnpnm" event={"ID":"25f6aa54-56b3-454a-81b6-fe8a56a31fb3","Type":"ContainerStarted","Data":"299d50227de930431d9eba2ef7815229303a94d03a5ea639dc97bd66854a1bab"}
Sep 10 23:38:07.073960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.073898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-jnpnm"
Sep 10 23:38:07.077585 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.077556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerStarted","Data":"48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b"}
Sep 10 23:38:07.077710 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.077621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerStarted","Data":"de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c"}
Sep 10 23:38:07.085954 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.085894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-jnpnm"
Sep 10 23:38:07.090959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.090116    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-jnpnm" podStartSLOduration=2.053435199 podStartE2EDuration="20.090095253s" totalImagesPullingTime="18.036660054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:07.088457301 +0000 UTC m=+129.044689106" watchObservedRunningTime="2026-09-10 23:38:07.090095253 +0000 UTC m=+129.046327057"
Sep 10 23:38:07.755958 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.755900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:07.854785 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:07.854743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:08.319163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:08.319114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:38:08.323410 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:08.322604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a62a794-f5a8-423e-bb4d-20ec9edccf4b-metrics-certs\") pod \"network-metrics-daemon-z6rwl\" (UID: \"4a62a794-f5a8-423e-bb4d-20ec9edccf4b\") " pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:38:08.345968 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:08.345936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xmh4\""
Sep 10 23:38:08.353899 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:08.353865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6rwl"
Sep 10 23:38:08.955917 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:08.955831    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z6rwl"]
Sep 10 23:38:09.122536 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:38:09.122496    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a62a794_f5a8_423e_bb4d_20ec9edccf4b.slice/crio-5ea5bcb90f5b4e060624dbe39bac149ee75b1243951a16a1239bd3feca27f452 WatchSource:0}: Error finding container 5ea5bcb90f5b4e060624dbe39bac149ee75b1243951a16a1239bd3feca27f452: Status 404 returned error can't find the container with id 5ea5bcb90f5b4e060624dbe39bac149ee75b1243951a16a1239bd3feca27f452
Sep 10 23:38:09.127447 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:09.127407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:09.228061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:09.228037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:10.104664 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:10.104617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerStarted","Data":"6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456"}
Sep 10 23:38:10.105375 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:10.104732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerStarted","Data":"a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c"}
Sep 10 23:38:10.105375 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:10.104780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerStarted","Data":"80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e"}
Sep 10 23:38:10.105375 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:10.104815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerStarted","Data":"8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b"}
Sep 10 23:38:10.106413 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:10.106386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z6rwl" event={"ID":"4a62a794-f5a8-423e-bb4d-20ec9edccf4b","Type":"ContainerStarted","Data":"5ea5bcb90f5b4e060624dbe39bac149ee75b1243951a16a1239bd3feca27f452"}
Sep 10 23:38:10.134630 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:10.134580    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.134659135 podStartE2EDuration="17.134560633s" totalImagesPullingTime="13.981102242s" totalInitContainerRuntime="18.799256ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:10.132206651 +0000 UTC m=+132.088438468" watchObservedRunningTime="2026-09-10 23:38:10.134560633 +0000 UTC m=+132.090792438"
Sep 10 23:38:11.113224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:11.113181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z6rwl" event={"ID":"4a62a794-f5a8-423e-bb4d-20ec9edccf4b","Type":"ContainerStarted","Data":"769daafe89ef0ebd6fd1a8dcca07c8916e3ea967f1651a546384e6573d230959"}
Sep 10 23:38:11.113696 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:11.113254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z6rwl" event={"ID":"4a62a794-f5a8-423e-bb4d-20ec9edccf4b","Type":"ContainerStarted","Data":"e9372af7f15f29d85c9d4d2b1293dcdf1290e59850bf8a791c8cb9497d8cfe38"}
Sep 10 23:38:11.132037 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:11.131990    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z6rwl" podStartSLOduration=130.560100694 podStartE2EDuration="2m12.131971069s" totalImagesPullingTime="1.571870375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:11.130306888 +0000 UTC m=+133.086538694" watchObservedRunningTime="2026-09-10 23:38:11.131971069 +0000 UTC m=+133.088202945"
Sep 10 23:38:11.223983 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:11.223943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:11.291905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:11.291873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:11.766725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:11.766682    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:38:11.767094 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:11.766753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:38:12.355268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:12.355232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:12.433926 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:12.433884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:13.247398 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:13.247359    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:13.348913 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:13.348874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:13.853813 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:13.853773    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:38:15.157042 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:15.157002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:15.257709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:15.257663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:15.357470 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:15.357433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:17.974324 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:17.974265    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw" podUID="2f4caf2f-2748-4852-afc6-19f7bf42c34e" containerName="registry" containerID="cri-o://d44d0d61a9d7b8493ba26731107f6bc03f935ae888c0130137ae418ac1e21085" gracePeriod=30
Sep 10 23:38:18.150058 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.150027    2578 generic.go:356] "Generic (PLEG): container finished" podID="2f4caf2f-2748-4852-afc6-19f7bf42c34e" containerID="d44d0d61a9d7b8493ba26731107f6bc03f935ae888c0130137ae418ac1e21085" exitCode=0
Sep 10 23:38:18.150229 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.150110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw" event={"ID":"2f4caf2f-2748-4852-afc6-19f7bf42c34e","Type":"ContainerDied","Data":"d44d0d61a9d7b8493ba26731107f6bc03f935ae888c0130137ae418ac1e21085"}
Sep 10 23:38:18.255709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.255638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:38:18.413570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413537    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f4caf2f-2748-4852-afc6-19f7bf42c34e-ca-trust-extracted\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.413741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.413741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-trusted-ca\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.413741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-certificates\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.413741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413653    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-image-registry-private-configuration\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.413741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-bound-sa-token\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.413741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg82z\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-kube-api-access-jg82z\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.413741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.413725    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-installation-pull-secrets\") pod \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\" (UID: \"2f4caf2f-2748-4852-afc6-19f7bf42c34e\") "
Sep 10 23:38:18.414089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.414068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:38:18.414497 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.414450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:38:18.416347 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.416318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:38:18.416515 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.416482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-kube-api-access-jg82z" (OuterVolumeSpecName: "kube-api-access-jg82z") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "kube-api-access-jg82z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:18.416672 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.416648    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:38:18.416731 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.416679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:18.416731 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.416683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:18.421990 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.421967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4caf2f-2748-4852-afc6-19f7bf42c34e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2f4caf2f-2748-4852-afc6-19f7bf42c34e" (UID: "2f4caf2f-2748-4852-afc6-19f7bf42c34e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:38:18.515328 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515297    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-installation-pull-secrets\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:18.515328 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515327    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f4caf2f-2748-4852-afc6-19f7bf42c34e-ca-trust-extracted\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:18.515507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515338    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:18.515507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515346    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-trusted-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:18.515507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515356    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f4caf2f-2748-4852-afc6-19f7bf42c34e-registry-certificates\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:18.515507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515366    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f4caf2f-2748-4852-afc6-19f7bf42c34e-image-registry-private-configuration\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:18.515507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515375    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-bound-sa-token\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:18.515507 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:18.515383    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg82z\" (UniqueName: \"kubernetes.io/projected/2f4caf2f-2748-4852-afc6-19f7bf42c34e-kube-api-access-jg82z\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:38:19.154940 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:19.154909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"
Sep 10 23:38:19.155376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:19.154912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7f6bf58d-tvcbw" event={"ID":"2f4caf2f-2748-4852-afc6-19f7bf42c34e","Type":"ContainerDied","Data":"df68e47f75c82807193607d0e3ee3fa137aa1aca98fe2adabc72c7b438744fd8"}
Sep 10 23:38:19.155376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:19.155043    2578 scope.go:117] "RemoveContainer" containerID="d44d0d61a9d7b8493ba26731107f6bc03f935ae888c0130137ae418ac1e21085"
Sep 10 23:38:19.174501 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:19.174477    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"]
Sep 10 23:38:19.178415 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:19.178392    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b7f6bf58d-tvcbw"]
Sep 10 23:38:20.533138 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:20.533103    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4caf2f-2748-4852-afc6-19f7bf42c34e" path="/var/lib/kubelet/pods/2f4caf2f-2748-4852-afc6-19f7bf42c34e/volumes"
Sep 10 23:38:27.189787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:27.189752    2578 generic.go:356] "Generic (PLEG): container finished" podID="01d9457a-ff0d-456e-b864-3204f0f384f0" containerID="b122cdb03fc99254853fa6d3b545c895443acfca721e77f85144e1186754e97f" exitCode=0
Sep 10 23:38:27.190181 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:27.189823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc" event={"ID":"01d9457a-ff0d-456e-b864-3204f0f384f0","Type":"ContainerDied","Data":"b122cdb03fc99254853fa6d3b545c895443acfca721e77f85144e1186754e97f"}
Sep 10 23:38:27.190231 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:27.190189    2578 scope.go:117] "RemoveContainer" containerID="b122cdb03fc99254853fa6d3b545c895443acfca721e77f85144e1186754e97f"
Sep 10 23:38:28.196003 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:28.195970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vdrjc" event={"ID":"01d9457a-ff0d-456e-b864-3204f0f384f0","Type":"ContainerStarted","Data":"20a7ffde94a18daa302c5ce96cd66f251388c9afe7ef7faa56d2cb9d10aa9e0f"}
Sep 10 23:38:28.243154 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:28.243121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:28.295213 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:28.295178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:31.771467 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:31.771437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:38:31.775077 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:31.775056    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78846476b6-cbm59"
Sep 10 23:38:36.436504 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:36.436462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:36.537581 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:36.537548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:36.638709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:36.638674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:53.853627 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:53.853541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:38:53.874554 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:53.874189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:38:54.313788 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:38:54.313761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:06.444896 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:06.444856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:06.643947 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:06.643905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:11.843794 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:11.843757    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:39:11.844742 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:11.844684    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="prometheus" containerID="cri-o://de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c" gracePeriod=600
Sep 10 23:39:11.844955 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:11.844915    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy" containerID="cri-o://a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c" gracePeriod=600
Sep 10 23:39:11.845121 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:11.844933    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="thanos-sidecar" containerID="cri-o://8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b" gracePeriod=600
Sep 10 23:39:11.845236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:11.844980    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456" gracePeriod=600
Sep 10 23:39:11.845236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:11.844980    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-web" containerID="cri-o://80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e" gracePeriod=600
Sep 10 23:39:11.845370 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:11.845000    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="config-reloader" containerID="cri-o://48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b" gracePeriod=600
Sep 10 23:39:12.375149 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375115    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerID="6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456" exitCode=0
Sep 10 23:39:12.375149 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375141    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerID="a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c" exitCode=0
Sep 10 23:39:12.375149 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375150    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerID="8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b" exitCode=0
Sep 10 23:39:12.375149 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375157    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerID="48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b" exitCode=0
Sep 10 23:39:12.375483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375164    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerID="de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c" exitCode=0
Sep 10 23:39:12.375483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456"}
Sep 10 23:39:12.375483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c"}
Sep 10 23:39:12.375483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b"}
Sep 10 23:39:12.375483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b"}
Sep 10 23:39:12.375483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:12.375344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c"}
Sep 10 23:39:13.083382 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.083356    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.158018 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.157983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-kubelet-serving-ca-bundle\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158018 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-tls-assets\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-grpc-tls\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-2\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158114    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtrjl\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-kube-api-access-vtrjl\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config-out\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-db\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-0\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158237    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-tls\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-web-config\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-metrics-client-ca\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-1\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-kube-rbac-proxy\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-thanos-prometheus-http-client-file\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-metrics-client-certs\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-serving-certs-ca-bundle\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158571    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.158704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-trusted-ca-bundle\") pod \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\" (UID: \"9fbb1027-4e0b-496c-bca7-e2af674f39d9\") "
Sep 10 23:39:13.159559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158875    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.159559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.158893    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.159559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.159214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:39:13.160029 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.159844    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:39:13.160921 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.160487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:39:13.160921 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.160649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:39:13.161103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.160935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.161103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.160966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:39:13.161582 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.161496    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:39:13.161582 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.161555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:39:13.162643 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.162599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.162874 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.162848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.164069 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.164039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.164165 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.164075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config" (OuterVolumeSpecName: "config") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.164230 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.164163    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.164230 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.164210    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.164397 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.164325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config-out" (OuterVolumeSpecName: "config-out") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:39:13.164562 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.164536    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.165957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.165936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-kube-api-access-vtrjl" (OuterVolumeSpecName: "kube-api-access-vtrjl") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "kube-api-access-vtrjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:39:13.174648 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.174627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-web-config" (OuterVolumeSpecName: "web-config") pod "9fbb1027-4e0b-496c-bca7-e2af674f39d9" (UID: "9fbb1027-4e0b-496c-bca7-e2af674f39d9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:39:13.259860 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259822    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.259860 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259853    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-web-config\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.259860 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259862    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-metrics-client-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259871    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259882    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-kube-rbac-proxy\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259891    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259901    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-metrics-client-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259910    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259918    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259927    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259935    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259942    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259951    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-tls-assets\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259959    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fbb1027-4e0b-496c-bca7-e2af674f39d9-secret-grpc-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259966    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrjl\" (UniqueName: \"kubernetes.io/projected/9fbb1027-4e0b-496c-bca7-e2af674f39d9-kube-api-access-vtrjl\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259974    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-config-out\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259981    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-db\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.260092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.259988    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fbb1027-4e0b-496c-bca7-e2af674f39d9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:39:13.382293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.382246    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerID="80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e" exitCode=0
Sep 10 23:39:13.382477 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.382304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e"}
Sep 10 23:39:13.382477 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.382345    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.382477 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.382389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fbb1027-4e0b-496c-bca7-e2af674f39d9","Type":"ContainerDied","Data":"404bd2051162fd5007f1910a87c13afffb00b0465ac9d0cac9a59f85673591fe"}
Sep 10 23:39:13.382477 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.382440    2578 scope.go:117] "RemoveContainer" containerID="6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456"
Sep 10 23:39:13.390033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.390016    2578 scope.go:117] "RemoveContainer" containerID="a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c"
Sep 10 23:39:13.398000 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.397983    2578 scope.go:117] "RemoveContainer" containerID="80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e"
Sep 10 23:39:13.404344 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.404328    2578 scope.go:117] "RemoveContainer" containerID="8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b"
Sep 10 23:39:13.407936 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.407915    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:39:13.411481 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.411460    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:39:13.412030 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.412016    2578 scope.go:117] "RemoveContainer" containerID="48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b"
Sep 10 23:39:13.418459 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.418435    2578 scope.go:117] "RemoveContainer" containerID="de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c"
Sep 10 23:39:13.425138 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.425122    2578 scope.go:117] "RemoveContainer" containerID="392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3"
Sep 10 23:39:13.433460 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.433426    2578 scope.go:117] "RemoveContainer" containerID="6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456"
Sep 10 23:39:13.433989 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:39:13.433952    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456\": container with ID starting with 6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456 not found: ID does not exist" containerID="6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456"
Sep 10 23:39:13.434092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.433999    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456"} err="failed to get container status \"6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456\": rpc error: code = NotFound desc = could not find container \"6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456\": container with ID starting with 6c454cefae47809cdbdafccedab0f666f2fd9dadac14804d85326deb0b04f456 not found: ID does not exist"
Sep 10 23:39:13.434092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.434042    2578 scope.go:117] "RemoveContainer" containerID="a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c"
Sep 10 23:39:13.435370 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:39:13.434643    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c\": container with ID starting with a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c not found: ID does not exist" containerID="a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c"
Sep 10 23:39:13.435370 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.434672    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c"} err="failed to get container status \"a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c\": rpc error: code = NotFound desc = could not find container \"a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c\": container with ID starting with a19767b51b96a6061006a0d615fd636bef6f40ef7a380c2d23589401ff19096c not found: ID does not exist"
Sep 10 23:39:13.435370 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.434691    2578 scope.go:117] "RemoveContainer" containerID="80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e"
Sep 10 23:39:13.435370 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:39:13.435001    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e\": container with ID starting with 80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e not found: ID does not exist" containerID="80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e"
Sep 10 23:39:13.435370 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.435041    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e"} err="failed to get container status \"80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e\": rpc error: code = NotFound desc = could not find container \"80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e\": container with ID starting with 80e067da3c062ef8d1b30ef1dd6a63a7a1d8cdc35f9dc12d8e92d396a2e0474e not found: ID does not exist"
Sep 10 23:39:13.435370 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.435062    2578 scope.go:117] "RemoveContainer" containerID="8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b"
Sep 10 23:39:13.435682 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:39:13.435381    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b\": container with ID starting with 8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b not found: ID does not exist" containerID="8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b"
Sep 10 23:39:13.435682 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.435407    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b"} err="failed to get container status \"8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b\": rpc error: code = NotFound desc = could not find container \"8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b\": container with ID starting with 8691ed71ab40af66a9e0a9f0a29f08897ca7d45cd9e16d60da348f4099c5a54b not found: ID does not exist"
Sep 10 23:39:13.435682 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.435427    2578 scope.go:117] "RemoveContainer" containerID="48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b"
Sep 10 23:39:13.435843 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:39:13.435822    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b\": container with ID starting with 48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b not found: ID does not exist" containerID="48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b"
Sep 10 23:39:13.435884 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.435851    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b"} err="failed to get container status \"48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b\": rpc error: code = NotFound desc = could not find container \"48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b\": container with ID starting with 48e01a0af7f293d667a9832b703d6502484fbb5942a3b8a98fa4cd09bc914b0b not found: ID does not exist"
Sep 10 23:39:13.435884 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.435866    2578 scope.go:117] "RemoveContainer" containerID="de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c"
Sep 10 23:39:13.436135 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:39:13.436117    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c\": container with ID starting with de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c not found: ID does not exist" containerID="de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c"
Sep 10 23:39:13.436199 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.436143    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c"} err="failed to get container status \"de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c\": rpc error: code = NotFound desc = could not find container \"de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c\": container with ID starting with de4e15e2be19ef3022f7d91d85f76d04e64a53fbf42475f2c583d2676196c31c not found: ID does not exist"
Sep 10 23:39:13.436199 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.436163    2578 scope.go:117] "RemoveContainer" containerID="392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3"
Sep 10 23:39:13.436474 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:39:13.436455    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3\": container with ID starting with 392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3 not found: ID does not exist" containerID="392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3"
Sep 10 23:39:13.436524 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.436480    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3"} err="failed to get container status \"392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3\": rpc error: code = NotFound desc = could not find container \"392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3\": container with ID starting with 392504eb7203acea8af177ccecd4450a24a6b6dbf6160606ee04d6f2986be4c3 not found: ID does not exist"
Sep 10 23:39:13.436739 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.436720    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:39:13.437074 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437062    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="config-reloader"
Sep 10 23:39:13.437109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437077    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="config-reloader"
Sep 10 23:39:13.437109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437085    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-web"
Sep 10 23:39:13.437109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437092    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-web"
Sep 10 23:39:13.437109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437102    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="init-config-reloader"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437110    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="init-config-reloader"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437128    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="thanos-sidecar"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437135    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="thanos-sidecar"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437143    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-thanos"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437151    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-thanos"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437159    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4caf2f-2748-4852-afc6-19f7bf42c34e" containerName="registry"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437164    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4caf2f-2748-4852-afc6-19f7bf42c34e" containerName="registry"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437170    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="prometheus"
Sep 10 23:39:13.437224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437185    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="prometheus"
Sep 10 23:39:13.437556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437229    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-thanos"
Sep 10 23:39:13.437556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437236    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="prometheus"
Sep 10 23:39:13.437556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437242    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy"
Sep 10 23:39:13.437556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437247    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="thanos-sidecar"
Sep 10 23:39:13.437556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437253    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4caf2f-2748-4852-afc6-19f7bf42c34e" containerName="registry"
Sep 10 23:39:13.437556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437259    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="config-reloader"
Sep 10 23:39:13.437556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.437265    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" containerName="kube-rbac-proxy-web"
Sep 10 23:39:13.442308 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.442264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.445278 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445257    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 23:39:13.445378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 23:39:13.445378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445304    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 23:39:13.445484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445472    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 23:39:13.445535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 23:39:13.445593 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kwjln\""
Sep 10 23:39:13.445735 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445718    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 23:39:13.445904 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.445887    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a081n4fej99kg\""
Sep 10 23:39:13.446031 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.446011    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 23:39:13.446092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.446071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 23:39:13.446153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.446089    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 23:39:13.446153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.446073    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 23:39:13.446378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.446357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 23:39:13.446571 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.446555    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 23:39:13.448762 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.448741    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 23:39:13.451212 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.451194    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 23:39:13.459581 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.459557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:39:13.563015 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.562979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563015 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1d0a207-3f3f-40b0-8e61-7470d99444ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t5b9\" (UniqueName: \"kubernetes.io/projected/e1d0a207-3f3f-40b0-8e61-7470d99444ee-kube-api-access-5t5b9\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-config\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1d0a207-3f3f-40b0-8e61-7470d99444ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.563569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.563455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.663941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.663907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.663941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.663942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.663958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.663981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1d0a207-3f3f-40b0-8e61-7470d99444ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.663999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t5b9\" (UniqueName: \"kubernetes.io/projected/e1d0a207-3f3f-40b0-8e61-7470d99444ee-kube-api-access-5t5b9\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-config\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1d0a207-3f3f-40b0-8e61-7470d99444ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.664787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.665229 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.665229 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.664964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.665559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.665534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.666146 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.666121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.667145 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.667111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1d0a207-3f3f-40b0-8e61-7470d99444ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.667145 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.667127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.667333 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.667199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.667333 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.667228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.667443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.667400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1d0a207-3f3f-40b0-8e61-7470d99444ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.667828 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.667754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.668059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.668039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.669755 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.669697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.669755 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.669703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-config\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.669871 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.669784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.669871 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.669818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.669977 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.669959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1d0a207-3f3f-40b0-8e61-7470d99444ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.670048 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.670032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1d0a207-3f3f-40b0-8e61-7470d99444ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.675131 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.675107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t5b9\" (UniqueName: \"kubernetes.io/projected/e1d0a207-3f3f-40b0-8e61-7470d99444ee-kube-api-access-5t5b9\") pod \"prometheus-k8s-0\" (UID: \"e1d0a207-3f3f-40b0-8e61-7470d99444ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:13.752070 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:13.752032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:14.030188 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:14.030148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:39:14.032959 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:39:14.032926    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d0a207_3f3f_40b0_8e61_7470d99444ee.slice/crio-cbb0dc2008ab2f1eca2975936163c5274a53294cf2e0178104429a89874fbd50 WatchSource:0}: Error finding container cbb0dc2008ab2f1eca2975936163c5274a53294cf2e0178104429a89874fbd50: Status 404 returned error can't find the container with id cbb0dc2008ab2f1eca2975936163c5274a53294cf2e0178104429a89874fbd50
Sep 10 23:39:14.389007 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:14.388970    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1d0a207-3f3f-40b0-8e61-7470d99444ee" containerID="e32bb7e9312f95e470c1f4466cf6a05d5ff52bda6823083e40b046c0e105b259" exitCode=0
Sep 10 23:39:14.389399 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:14.389035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerDied","Data":"e32bb7e9312f95e470c1f4466cf6a05d5ff52bda6823083e40b046c0e105b259"}
Sep 10 23:39:14.389399 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:14.389103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerStarted","Data":"cbb0dc2008ab2f1eca2975936163c5274a53294cf2e0178104429a89874fbd50"}
Sep 10 23:39:14.506486 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:14.506369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:14.534293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:14.534185    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fbb1027-4e0b-496c-bca7-e2af674f39d9" path="/var/lib/kubelet/pods/9fbb1027-4e0b-496c-bca7-e2af674f39d9/volumes"
Sep 10 23:39:15.396635 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:15.396595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerStarted","Data":"e00c52a329825e1f03c3527e28db18af5fa105916261be68dd9ee853870e90a5"}
Sep 10 23:39:15.397021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:15.396678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerStarted","Data":"06528cf18a5039fe202656c0a00224d8bd25006bc068db92459b9ebe6912b58d"}
Sep 10 23:39:15.397021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:15.396754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerStarted","Data":"4b5f8e2cf7c65be300dad21c7403b79dc90e498c65777f24d0c326f809e0e325"}
Sep 10 23:39:15.397021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:15.396793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerStarted","Data":"2d20fde680888c532b05443ce2745763978edf3d3adfa62c0f8830e535551b83"}
Sep 10 23:39:15.397021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:15.396836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerStarted","Data":"a38e2d8bfd53d063ad23baac96b816bff1506bc4f6e2782a9f104a2145c26699"}
Sep 10 23:39:15.397021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:15.396874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1d0a207-3f3f-40b0-8e61-7470d99444ee","Type":"ContainerStarted","Data":"ce74f0800f3bcedfb2c06c43b8b686bd3a4196964d8f1d9ee1922d63cd923358"}
Sep 10 23:39:15.426257 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:15.426198    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.403769027 podStartE2EDuration="2.426180011s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.410984ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:39:15.424167784 +0000 UTC m=+197.380399608" watchObservedRunningTime="2026-09-10 23:39:15.426180011 +0000 UTC m=+197.382411813"
Sep 10 23:39:18.752369 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:18.752333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:39:25.552887 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:25.552844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:25.611138 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:25.611107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:25.660376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:25.660348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:54.882344 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:39:54.882306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:11.081857 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:11.081824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:13.753079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:13.753045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:40:13.768262 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:13.768237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:40:14.637084 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:14.637055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:40:21.683535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:21.683452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:21.731456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:21.731424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:33.708101 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:33.708058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:33.808862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:33.808834    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:43.246801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:43.246761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:43.347330 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:43.347266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:47.704979 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:47.704946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:48.020142 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:48.020104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:48.830634 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:48.830603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:48.890921 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:48.890886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:58.480941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:58.480907    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:40:58.486621 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:58.486597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:40:58.492179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:58.492160    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 23:40:58.936964 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:58.936934    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:59.037832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:59.037796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:59.093112 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:40:59.093086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:25.298795 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:25.298754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:27.165941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:27.165909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:47.124013 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:47.123973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:47.942686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:47.942653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:50.747639 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:50.747598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:50.839733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:50.839686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:51.189521 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:51.189491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:52.021749 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:52.021709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:52.235070 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:52.235034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:52.612171 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:52.612133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:52.809121 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:52.809085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:54.383499 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:54.383468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:54.549991 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:54.549957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:57.280704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:57.280666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:57.339311 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:41:57.339258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:02.116981 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:02.116944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:15.814743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:15.814707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:15.911879 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:15.911843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:19.470326 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:19.470269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:19.523445 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:19.523415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:21.522905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:21.522867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:21.636819 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:21.636787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:23.500887 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:23.500846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:27.768029 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.767995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:27.837335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.837299    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"]
Sep 10 23:42:27.840427 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.840409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.846420 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.846396    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 23:42:27.847588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.847561    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 23:42:27.847689 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.847610    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 23:42:27.847689 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.847631    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:42:27.847689 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.847674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 23:42:27.848089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.848071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tswhv\""
Sep 10 23:42:27.869262 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.869232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"]
Sep 10 23:42:27.882328 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.882308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/777661a7-5930-4dbd-a1d2-6db42f470a23-metrics-cert\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.882440 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.882336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjg6\" (UniqueName: \"kubernetes.io/projected/777661a7-5930-4dbd-a1d2-6db42f470a23-kube-api-access-nfjg6\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.882440 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.882355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/777661a7-5930-4dbd-a1d2-6db42f470a23-cert\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.882440 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.882375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/777661a7-5930-4dbd-a1d2-6db42f470a23-manager-config\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.961130 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.961100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:27.983153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.983129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/777661a7-5930-4dbd-a1d2-6db42f470a23-metrics-cert\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.983268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.983158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjg6\" (UniqueName: \"kubernetes.io/projected/777661a7-5930-4dbd-a1d2-6db42f470a23-kube-api-access-nfjg6\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.983268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.983179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/777661a7-5930-4dbd-a1d2-6db42f470a23-cert\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.983268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.983237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/777661a7-5930-4dbd-a1d2-6db42f470a23-manager-config\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.983939 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.983920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/777661a7-5930-4dbd-a1d2-6db42f470a23-manager-config\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.985522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.985503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/777661a7-5930-4dbd-a1d2-6db42f470a23-cert\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:27.985609 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:27.985557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/777661a7-5930-4dbd-a1d2-6db42f470a23-metrics-cert\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:28.005098 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:28.005077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjg6\" (UniqueName: \"kubernetes.io/projected/777661a7-5930-4dbd-a1d2-6db42f470a23-kube-api-access-nfjg6\") pod \"lws-controller-manager-98d67b8cf-hrlhr\" (UID: \"777661a7-5930-4dbd-a1d2-6db42f470a23\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:28.154230 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:28.154147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:28.429665 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:28.429589    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"]
Sep 10 23:42:28.432736 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:42:28.432702    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777661a7_5930_4dbd_a1d2_6db42f470a23.slice/crio-f87e691e43cce097bbc58dda0046fe5e5f399c0d825602e5e7d32debcd92aa5c WatchSource:0}: Error finding container f87e691e43cce097bbc58dda0046fe5e5f399c0d825602e5e7d32debcd92aa5c: Status 404 returned error can't find the container with id f87e691e43cce097bbc58dda0046fe5e5f399c0d825602e5e7d32debcd92aa5c
Sep 10 23:42:28.434577 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:28.434562    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:42:29.125533 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:29.125489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr" event={"ID":"777661a7-5930-4dbd-a1d2-6db42f470a23","Type":"ContainerStarted","Data":"f87e691e43cce097bbc58dda0046fe5e5f399c0d825602e5e7d32debcd92aa5c"}
Sep 10 23:42:31.135262 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:31.135226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr" event={"ID":"777661a7-5930-4dbd-a1d2-6db42f470a23","Type":"ContainerStarted","Data":"5f31ff353dbb700e49842fbba9f2241f9cc92fedac622750f2f4b0e8869fe35a"}
Sep 10 23:42:31.135725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:31.135385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:31.155359 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:31.155320    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr" podStartSLOduration=1.998272008 podStartE2EDuration="4.155307062s" totalImagesPullingTime="2.157035054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:31.153808876 +0000 UTC m=+393.110040679" watchObservedRunningTime="2026-09-10 23:42:31.155307062 +0000 UTC m=+393.111538865"
Sep 10 23:42:37.906170 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:37.906131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:38.002170 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.002135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:38.539417 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.539381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:38.735823 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.735787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:38.844330 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.844248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:38.892830 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.892798    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"]
Sep 10 23:42:38.896352 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.896326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:38.899867 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.899845    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 23:42:38.900386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.900363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 23:42:38.900386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.900380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 23:42:38.900569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.900363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zkqzc\""
Sep 10 23:42:38.900673 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.900656    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 23:42:38.915195 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.915167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"]
Sep 10 23:42:38.968800 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.968764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f0c-09a7-4f3a-8331-6812961f6256-manifests\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:38.968969 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.968809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ngw\" (UniqueName: \"kubernetes.io/projected/99ec7f0c-09a7-4f3a-8331-6812961f6256-kube-api-access-w7ngw\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:38.968969 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.968840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99ec7f0c-09a7-4f3a-8331-6812961f6256-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:38.968969 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.968895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ec7f0c-09a7-4f3a-8331-6812961f6256-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:38.969072 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.968973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f0c-09a7-4f3a-8331-6812961f6256-tmp\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:38.996626 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:38.996595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:39.069406 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.069369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ec7f0c-09a7-4f3a-8331-6812961f6256-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.069564 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.069439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f0c-09a7-4f3a-8331-6812961f6256-tmp\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.069564 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.069488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f0c-09a7-4f3a-8331-6812961f6256-manifests\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.069564 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.069510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ngw\" (UniqueName: \"kubernetes.io/projected/99ec7f0c-09a7-4f3a-8331-6812961f6256-kube-api-access-w7ngw\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.069564 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.069541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99ec7f0c-09a7-4f3a-8331-6812961f6256-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.070025 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.069994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f0c-09a7-4f3a-8331-6812961f6256-manifests\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.071847 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.071810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f0c-09a7-4f3a-8331-6812961f6256-tmp\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.071963 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.071872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ec7f0c-09a7-4f3a-8331-6812961f6256-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.072032 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.071972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99ec7f0c-09a7-4f3a-8331-6812961f6256-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.078051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.078028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ngw\" (UniqueName: \"kubernetes.io/projected/99ec7f0c-09a7-4f3a-8331-6812961f6256-kube-api-access-w7ngw\") pod \"opendatahub-operator-controller-manager-5bb6f54779-ktb4q\" (UID: \"99ec7f0c-09a7-4f3a-8331-6812961f6256\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.100827 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.100739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:39.199622 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.199570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:39.207233 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.207205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:39.265300 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.264774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:39.331332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.331291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:39.483599 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:39.483576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"]
Sep 10 23:42:39.486019 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:42:39.485994    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ec7f0c_09a7_4f3a_8331_6812961f6256.slice/crio-5b01a5ac3c51ac95b4421b15007553557fa0b44a1f3fcebe349b3d3932740897 WatchSource:0}: Error finding container 5b01a5ac3c51ac95b4421b15007553557fa0b44a1f3fcebe349b3d3932740897: Status 404 returned error can't find the container with id 5b01a5ac3c51ac95b4421b15007553557fa0b44a1f3fcebe349b3d3932740897
Sep 10 23:42:40.174411 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:40.174124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q" event={"ID":"99ec7f0c-09a7-4f3a-8331-6812961f6256","Type":"ContainerStarted","Data":"5b01a5ac3c51ac95b4421b15007553557fa0b44a1f3fcebe349b3d3932740897"}
Sep 10 23:42:42.142494 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:42.142463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-hrlhr"
Sep 10 23:42:43.189179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:43.189141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q" event={"ID":"99ec7f0c-09a7-4f3a-8331-6812961f6256","Type":"ContainerStarted","Data":"0ab2d4483b14e12d00cf90f10e6eda3a08d420848ae2ae4fd8f76f4ffbb89720"}
Sep 10 23:42:44.194809 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:44.194770    2578 generic.go:356] "Generic (PLEG): container finished" podID="99ec7f0c-09a7-4f3a-8331-6812961f6256" containerID="0ab2d4483b14e12d00cf90f10e6eda3a08d420848ae2ae4fd8f76f4ffbb89720" exitCode=0
Sep 10 23:42:44.195178 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:44.194820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q" event={"ID":"99ec7f0c-09a7-4f3a-8331-6812961f6256","Type":"ContainerDied","Data":"0ab2d4483b14e12d00cf90f10e6eda3a08d420848ae2ae4fd8f76f4ffbb89720"}
Sep 10 23:42:45.201083 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:45.201046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q" event={"ID":"99ec7f0c-09a7-4f3a-8331-6812961f6256","Type":"ContainerStarted","Data":"65d766e61035a27feebb456b9a698be42130dd72e2c0a3b8af4d25efc0b5b787"}
Sep 10 23:42:45.201530 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:45.201200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:45.223955 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:45.223913    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q" podStartSLOduration=2.887779201 podStartE2EDuration="7.223901854s" totalImagesPullingTime="3.266033216s" totalInitContainerRuntime="1.070089437s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:45.222781304 +0000 UTC m=+407.179013108" watchObservedRunningTime="2026-09-10 23:42:45.223901854 +0000 UTC m=+407.180133657"
Sep 10 23:42:51.212223 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:51.212183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:51.623862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:51.623828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:51.675361 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:51.675334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:56.208140 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:56.208108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-ktb4q"
Sep 10 23:42:57.879197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:57.879161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:58.021397 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.021361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:58.322385 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.322343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:58.429252 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.429224    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"]
Sep 10 23:42:58.438367 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.438346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.440909 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.440881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 23:42:58.441042 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.440917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 23:42:58.441042 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.440939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kcqzk\""
Sep 10 23:42:58.447595 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.447567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"]
Sep 10 23:42:58.532636 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.532608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb639d0-026a-4a2e-a501-2a7e921288f5-tls-certs\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.532636 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.532635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldttr\" (UniqueName: \"kubernetes.io/projected/0bb639d0-026a-4a2e-a501-2a7e921288f5-kube-api-access-ldttr\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.532789 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.532708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb639d0-026a-4a2e-a501-2a7e921288f5-tmp\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.628092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.628008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:58.633081 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.633057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb639d0-026a-4a2e-a501-2a7e921288f5-tls-certs\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.633197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.633087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldttr\" (UniqueName: \"kubernetes.io/projected/0bb639d0-026a-4a2e-a501-2a7e921288f5-kube-api-access-ldttr\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.633197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.633118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb639d0-026a-4a2e-a501-2a7e921288f5-tmp\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.635286 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.635247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb639d0-026a-4a2e-a501-2a7e921288f5-tmp\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.635423 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.635405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb639d0-026a-4a2e-a501-2a7e921288f5-tls-certs\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.641373 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.641352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldttr\" (UniqueName: \"kubernetes.io/projected/0bb639d0-026a-4a2e-a501-2a7e921288f5-kube-api-access-ldttr\") pod \"kube-auth-proxy-547c6fc866-zjtsj\" (UID: \"0bb639d0-026a-4a2e-a501-2a7e921288f5\") " pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:58.749221 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:58.749196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"
Sep 10 23:42:59.021823 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:59.021795    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj"]
Sep 10 23:42:59.024119 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:42:59.024095    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb639d0_026a_4a2e_a501_2a7e921288f5.slice/crio-d66f35847a37547f207a1dcc74beeb5ab7bcca1c58fc0393170f2a9455978c77 WatchSource:0}: Error finding container d66f35847a37547f207a1dcc74beeb5ab7bcca1c58fc0393170f2a9455978c77: Status 404 returned error can't find the container with id d66f35847a37547f207a1dcc74beeb5ab7bcca1c58fc0393170f2a9455978c77
Sep 10 23:42:59.264099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:59.264067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj" event={"ID":"0bb639d0-026a-4a2e-a501-2a7e921288f5","Type":"ContainerStarted","Data":"d66f35847a37547f207a1dcc74beeb5ab7bcca1c58fc0393170f2a9455978c77"}
Sep 10 23:42:59.620068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:59.620020    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:59.716871 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:42:59.716810    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:00.496802 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:00.496762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.020331 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.020289    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.074649 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.074614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.158588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.158543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.207067 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.207030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.280095 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.279996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj" event={"ID":"0bb639d0-026a-4a2e-a501-2a7e921288f5","Type":"ContainerStarted","Data":"3ec1562dd2012b3ae9dc4f7592f4152f2f5cdafe5a22f2e57d31d0db32f5ca17"}
Sep 10 23:43:02.297417 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.297370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-547c6fc866-zjtsj" podStartSLOduration=1.69738969 podStartE2EDuration="4.297355772s" totalImagesPullingTime="2.599966082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:02.29549173 +0000 UTC m=+424.251723534" watchObservedRunningTime="2026-09-10 23:43:02.297355772 +0000 UTC m=+424.253587575"
Sep 10 23:43:02.757746 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.757659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.813126 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:02.813090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:08.827972 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:08.827934    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:08.928824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:08.928795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:09.029347 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:09.029309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:09.124150 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:09.124074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:09.419156 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:09.419056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:12.586752 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:12.586713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:12.688588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:12.688552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:12.740475 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:12.740446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:13.923255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:13.923220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:14.193465 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:14.193376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:14.294480 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:14.294448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:14.482815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:14.482736    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:14.617409 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:14.617378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:14.783543 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:14.783511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:19.947753 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:19.947704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:20.109147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:20.109111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:20.262261 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:20.262215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:20.405085 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:20.405059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:21.550122 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:21.550044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:21.758629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:21.758587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:28.390332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:28.390297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:28.510592 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:43:28.510554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:10.015946 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:10.015888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:10.114622 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:10.114585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:23.405208 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:23.405169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:26.018540 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:26.018499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:26.406744 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:26.406666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:26.812099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:26.812062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:29.688654 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:29.688606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:29.858393 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:29.858353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:30.410120 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:30.410084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:30.686265 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:30.686173    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:30.839459 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:30.839418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:32.353261 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:32.353226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:33.013253 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:33.013218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:33.638802 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:33.638762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:36.014754 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:36.014717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:36.119402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:36.119375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:36.559940 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:36.559904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:36.847617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:36.847535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:39.799875 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:39.799839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:39.902053 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:39.902008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:40.215431 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:40.215341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:40.322457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:40.322409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:40.613808 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:40.613771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:40.714717 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:40.714680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:47.696132 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:47.696093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:47.800136 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:47.800097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:48.292709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.292670    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"]
Sep 10 23:44:48.296259 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.296241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.298776 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.298756    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fwgcp\""
Sep 10 23:44:48.298857 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.298758    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 23:44:48.298857 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.298781    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 23:44:48.307335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.307315    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"]
Sep 10 23:44:48.445420 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.445385    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:48.451410 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.451384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8600967b-cbb0-4d07-87e0-fa266b3c4f11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l58rd\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.451495 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.451434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcj2k\" (UniqueName: \"kubernetes.io/projected/8600967b-cbb0-4d07-87e0-fa266b3c4f11-kube-api-access-fcj2k\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l58rd\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.552579 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.552495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8600967b-cbb0-4d07-87e0-fa266b3c4f11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l58rd\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.552579 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.552550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcj2k\" (UniqueName: \"kubernetes.io/projected/8600967b-cbb0-4d07-87e0-fa266b3c4f11-kube-api-access-fcj2k\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l58rd\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.552892 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.552869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8600967b-cbb0-4d07-87e0-fa266b3c4f11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l58rd\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.562130 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.562106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcj2k\" (UniqueName: \"kubernetes.io/projected/8600967b-cbb0-4d07-87e0-fa266b3c4f11-kube-api-access-fcj2k\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l58rd\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.563160 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.563132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:48.607105 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.607083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:48.904317 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:48.904227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"]
Sep 10 23:44:48.911625 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:44:48.911598    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8600967b_cbb0_4d07_87e0_fa266b3c4f11.slice/crio-24a80d558165ef815f02fee626c1273ef6fa9ec3ea8f573b13349d075a4d9381 WatchSource:0}: Error finding container 24a80d558165ef815f02fee626c1273ef6fa9ec3ea8f573b13349d075a4d9381: Status 404 returned error can't find the container with id 24a80d558165ef815f02fee626c1273ef6fa9ec3ea8f573b13349d075a4d9381
Sep 10 23:44:49.743595 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:49.743557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd" event={"ID":"8600967b-cbb0-4d07-87e0-fa266b3c4f11","Type":"ContainerStarted","Data":"24a80d558165ef815f02fee626c1273ef6fa9ec3ea8f573b13349d075a4d9381"}
Sep 10 23:44:50.890219 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:50.890179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:50.991171 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:50.991095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:53.764560 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:53.764529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd" event={"ID":"8600967b-cbb0-4d07-87e0-fa266b3c4f11","Type":"ContainerStarted","Data":"58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd"}
Sep 10 23:44:53.764966 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:53.764681    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:44:53.786570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:53.786527    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd" podStartSLOduration=1.6492817020000001 podStartE2EDuration="5.786512535s" totalImagesPullingTime="4.137230833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:44:53.78475148 +0000 UTC m=+535.740983288" watchObservedRunningTime="2026-09-10 23:44:53.786512535 +0000 UTC m=+535.742744338"
Sep 10 23:44:55.226005 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:55.225969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:55.325461 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:55.325427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:59.027154 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:59.027122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:59.094780 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:59.094716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:59.224930 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:59.224890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:59.331484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:44:59.331397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:00.225238 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:00.225202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:00.280060 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:00.280023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:04.772021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:04.771984    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 10 23:45:09.436005 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:09.435963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:09.536379 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:09.536342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:11.874447 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:11.874410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:11.975394 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:11.975355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:21.241981 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:21.241943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:21.562666 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:21.562629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:23.326763 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.326721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:23.437814 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.437778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:23.492179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.492138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:23.689466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.689387    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:45:23.692760 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.692744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:23.695505 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.695485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 23:45:23.695622 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.695505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tdvbq\""
Sep 10 23:45:23.701243 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.701219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:45:23.725288 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.725239    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:45:23.744039 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.744010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnwc9\" (UniqueName: \"kubernetes.io/projected/02bda458-d06e-4447-be8c-549cc03058b0-kube-api-access-bnwc9\") pod \"limitador-limitador-84994f7ffc-bkvpx\" (UID: \"02bda458-d06e-4447-be8c-549cc03058b0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:23.744164 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.744081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/02bda458-d06e-4447-be8c-549cc03058b0-config-file\") pod \"limitador-limitador-84994f7ffc-bkvpx\" (UID: \"02bda458-d06e-4447-be8c-549cc03058b0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:23.844721 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.844686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/02bda458-d06e-4447-be8c-549cc03058b0-config-file\") pod \"limitador-limitador-84994f7ffc-bkvpx\" (UID: \"02bda458-d06e-4447-be8c-549cc03058b0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:23.844901 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.844739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnwc9\" (UniqueName: \"kubernetes.io/projected/02bda458-d06e-4447-be8c-549cc03058b0-kube-api-access-bnwc9\") pod \"limitador-limitador-84994f7ffc-bkvpx\" (UID: \"02bda458-d06e-4447-be8c-549cc03058b0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:23.845399 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.845379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/02bda458-d06e-4447-be8c-549cc03058b0-config-file\") pod \"limitador-limitador-84994f7ffc-bkvpx\" (UID: \"02bda458-d06e-4447-be8c-549cc03058b0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:23.852695 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:23.852671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnwc9\" (UniqueName: \"kubernetes.io/projected/02bda458-d06e-4447-be8c-549cc03058b0-kube-api-access-bnwc9\") pod \"limitador-limitador-84994f7ffc-bkvpx\" (UID: \"02bda458-d06e-4447-be8c-549cc03058b0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:24.004448 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:24.004364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:24.154199 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:24.154160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:24.251072 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:24.251024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:24.275673 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:24.275584    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:45:24.278575 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:45:24.278547    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bda458_d06e_4447_be8c_549cc03058b0.slice/crio-208103dc89d909fc10ab8ff9b18819e8938f2c70c464090121263655adde25ed WatchSource:0}: Error finding container 208103dc89d909fc10ab8ff9b18819e8938f2c70c464090121263655adde25ed: Status 404 returned error can't find the container with id 208103dc89d909fc10ab8ff9b18819e8938f2c70c464090121263655adde25ed
Sep 10 23:45:24.908727 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:24.908693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx" event={"ID":"02bda458-d06e-4447-be8c-549cc03058b0","Type":"ContainerStarted","Data":"208103dc89d909fc10ab8ff9b18819e8938f2c70c464090121263655adde25ed"}
Sep 10 23:45:25.915058 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:25.915021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx" event={"ID":"02bda458-d06e-4447-be8c-549cc03058b0","Type":"ContainerStarted","Data":"6ef94c8d42b4e6e5d060154a1dbc9fb9c7c5fa05e6936b0a77e663fd12859fbf"}
Sep 10 23:45:25.915520 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:25.915089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:25.930428 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:25.930340    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx" podStartSLOduration=1.735687141 podStartE2EDuration="2.930321785s" totalImagesPullingTime="1.194634644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:45:25.929631943 +0000 UTC m=+567.885863745" watchObservedRunningTime="2026-09-10 23:45:25.930321785 +0000 UTC m=+567.886553588"
Sep 10 23:45:36.919838 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:36.919801    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"
Sep 10 23:45:39.453840 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:39.453807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:39.553112 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:39.553066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:39.992725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:39.992692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.786540 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:50.786496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:51.109644 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.109556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:51.176809 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.176780    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dzffp"]
Sep 10 23:45:51.180365 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.180344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:51.184220 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.184203    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bp8fw\""
Sep 10 23:45:51.184314 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.184206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 23:45:51.187536 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.187513    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dzffp"]
Sep 10 23:45:51.280034 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.280003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-tls-cert\") pod \"authorino-6d4dfc649f-dzffp\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") " pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:51.280176 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.280041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgknn\" (UniqueName: \"kubernetes.io/projected/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-kube-api-access-hgknn\") pod \"authorino-6d4dfc649f-dzffp\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") " pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:51.321354 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.321324    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dzffp"]
Sep 10 23:45:51.321541 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:45:51.321522    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hgknn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-dzffp" podUID="1753d4eb-7d4b-4acd-b2b7-f8849ec76895"
Sep 10 23:45:51.380817 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.380755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-tls-cert\") pod \"authorino-6d4dfc649f-dzffp\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") " pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:51.380817 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.380787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgknn\" (UniqueName: \"kubernetes.io/projected/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-kube-api-access-hgknn\") pod \"authorino-6d4dfc649f-dzffp\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") " pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:51.383186 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.383156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-tls-cert\") pod \"authorino-6d4dfc649f-dzffp\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") " pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:51.388752 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:51.388730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgknn\" (UniqueName: \"kubernetes.io/projected/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-kube-api-access-hgknn\") pod \"authorino-6d4dfc649f-dzffp\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") " pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:52.027572 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.027540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:52.034497 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.034478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:52.085144 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.085118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-tls-cert\") pod \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") "
Sep 10 23:45:52.085248 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.085171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgknn\" (UniqueName: \"kubernetes.io/projected/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-kube-api-access-hgknn\") pod \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\" (UID: \"1753d4eb-7d4b-4acd-b2b7-f8849ec76895\") "
Sep 10 23:45:52.087222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.087200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1753d4eb-7d4b-4acd-b2b7-f8849ec76895" (UID: "1753d4eb-7d4b-4acd-b2b7-f8849ec76895"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:45:52.087334 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.087304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-kube-api-access-hgknn" (OuterVolumeSpecName: "kube-api-access-hgknn") pod "1753d4eb-7d4b-4acd-b2b7-f8849ec76895" (UID: "1753d4eb-7d4b-4acd-b2b7-f8849ec76895"). InnerVolumeSpecName "kube-api-access-hgknn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:45:52.186142 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.186121    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-tls-cert\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:45:52.186142 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:52.186143    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgknn\" (UniqueName: \"kubernetes.io/projected/1753d4eb-7d4b-4acd-b2b7-f8849ec76895-kube-api-access-hgknn\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:45:53.031872 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:53.031843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dzffp"
Sep 10 23:45:53.057980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:53.057954    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dzffp"]
Sep 10 23:45:53.062197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:53.062171    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dzffp"]
Sep 10 23:45:54.533253 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:54.533215    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1753d4eb-7d4b-4acd-b2b7-f8849ec76895" path="/var/lib/kubelet/pods/1753d4eb-7d4b-4acd-b2b7-f8849ec76895/volumes"
Sep 10 23:45:58.510957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:58.510918    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:45:58.513185 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:58.513162    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:45:58.905167 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:58.905133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:59.001113 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:59.000998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:59.101317 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:59.101269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:59.396451 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:45:59.396412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:02.982886 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:02.982850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:03.080360 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:03.080311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:19.539027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:19.538984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:28.239119 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:28.238981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:29.638435 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:29.638400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:34.179125 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:34.179083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:34.279034 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:34.279003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:35.646095 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:35.646060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:35.734226 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:35.734182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:36.034079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:36.034043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:36.137761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:36.137719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:36.235662 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:36.235628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:36.260403 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:36.260371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:46:36.523363 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:36.523329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:48.809752 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:48.809715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:48.902468 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:46:48.902434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:04.273413 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:04.273380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:04.796392 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:04.796345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:47:19.959284 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:19.959240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:20.864591 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:20.864558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:21.489017 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:21.488980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:47:30.479679 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:30.479638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:31.382129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:31.382094    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:47:33.532167 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.532134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"]
Sep 10 23:47:33.535787 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.535761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.538466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.538443    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 23:47:33.538642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.538462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 23:47:33.539737 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.539708    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 23:47:33.539847 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.539782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ds5n5\""
Sep 10 23:47:33.545433 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.545406    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"]
Sep 10 23:47:33.547682 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.547653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faac05e-1552-4184-8c2e-4b3f5eb44340-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.547865 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.547843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.547980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.547922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.547980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.547970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.548095 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.547993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.548095 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.548046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmt9\" (UniqueName: \"kubernetes.io/projected/3faac05e-1552-4184-8c2e-4b3f5eb44340-kube-api-access-2vmt9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.576873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.576841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:33.648509 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.648473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faac05e-1552-4184-8c2e-4b3f5eb44340-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.648509 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.648512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.648728 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.648546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.648728 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.648594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.648728 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.648620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.648728 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.648649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmt9\" (UniqueName: \"kubernetes.io/projected/3faac05e-1552-4184-8c2e-4b3f5eb44340-kube-api-access-2vmt9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.649009 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.648977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.649099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.649019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.649099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.649058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.650927 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.650903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3faac05e-1552-4184-8c2e-4b3f5eb44340-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.651051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.651036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3faac05e-1552-4184-8c2e-4b3f5eb44340-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.656394 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.656374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmt9\" (UniqueName: \"kubernetes.io/projected/3faac05e-1552-4184-8c2e-4b3f5eb44340-kube-api-access-2vmt9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cncz\" (UID: \"3faac05e-1552-4184-8c2e-4b3f5eb44340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:33.848424 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:33.848335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:34.117452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:34.117425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"]
Sep 10 23:47:34.120120 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:47:34.120094    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faac05e_1552_4184_8c2e_4b3f5eb44340.slice/crio-383255e3c507016f12412d714eaec31cbdad8bc7989a38f0e6b2552cae5e4006 WatchSource:0}: Error finding container 383255e3c507016f12412d714eaec31cbdad8bc7989a38f0e6b2552cae5e4006: Status 404 returned error can't find the container with id 383255e3c507016f12412d714eaec31cbdad8bc7989a38f0e6b2552cae5e4006
Sep 10 23:47:34.121861 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:34.121841    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:47:34.487066 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:34.486964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz" event={"ID":"3faac05e-1552-4184-8c2e-4b3f5eb44340","Type":"ContainerStarted","Data":"383255e3c507016f12412d714eaec31cbdad8bc7989a38f0e6b2552cae5e4006"}
Sep 10 23:47:34.694342 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:34.694306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:47:35.494266 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:35.494181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz" event={"ID":"3faac05e-1552-4184-8c2e-4b3f5eb44340","Type":"ContainerStarted","Data":"2073f24783b1b16bd0453601fa81b98550653c278747f33f6c2d1b524d99758a"}
Sep 10 23:47:35.511501 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:35.511456    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz" podStartSLOduration=1.495754412 podStartE2EDuration="2.511440673s" totalImagesPullingTime="1.015686261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:35.510696066 +0000 UTC m=+697.466927870" watchObservedRunningTime="2026-09-10 23:47:35.511440673 +0000 UTC m=+697.467672498"
Sep 10 23:47:40.965516 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:40.965482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:42.389201 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:42.389167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:47:43.848815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:43.848772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:43.849201 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:43.848837    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:43.861411 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:43.861390    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:44.550957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:44.550924    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cncz"
Sep 10 23:47:52.581892 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:52.581810    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:52.634360 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:52.634331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:55.453828 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:55.453784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:57.088438 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:57.088398    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:47:59.075261 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:59.075224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:59.126872 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:47:59.126836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:48:14.930384 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:14.930351    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"]
Sep 10 23:48:14.935812 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:14.935788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:14.938957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:14.938934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 23:48:14.951643 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:14.951600    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"]
Sep 10 23:48:14.962087 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:14.962052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:48:15.082331 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.082295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.082517 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.082370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.082517 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.082400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.082517 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.082430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7fm\" (UniqueName: \"kubernetes.io/projected/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-kube-api-access-xq7fm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.082636 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.082508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.082636 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.082538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.183702 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.183625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.183702 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.183667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.183702 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.183688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.183980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.183708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7fm\" (UniqueName: \"kubernetes.io/projected/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-kube-api-access-xq7fm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.183980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.183742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.183980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.183765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.184136 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.184107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.184190 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.184157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.184237 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.184184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.186109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.186087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.186227 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.186152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.192134 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.192109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7fm\" (UniqueName: \"kubernetes.io/projected/a21186bb-54f4-4c9a-baeb-30e0e21ea30d-kube-api-access-xq7fm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f582d\" (UID: \"a21186bb-54f4-4c9a-baeb-30e0e21ea30d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.263674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.263655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:15.531437 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.531410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"]
Sep 10 23:48:15.533731 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:48:15.533706    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda21186bb_54f4_4c9a_baeb_30e0e21ea30d.slice/crio-c69776f070416e6abb2e6eddb400c0d11fc82a43928df636e6439ce188071e14 WatchSource:0}: Error finding container c69776f070416e6abb2e6eddb400c0d11fc82a43928df636e6439ce188071e14: Status 404 returned error can't find the container with id c69776f070416e6abb2e6eddb400c0d11fc82a43928df636e6439ce188071e14
Sep 10 23:48:15.680166 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:15.680131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d" event={"ID":"a21186bb-54f4-4c9a-baeb-30e0e21ea30d","Type":"ContainerStarted","Data":"c69776f070416e6abb2e6eddb400c0d11fc82a43928df636e6439ce188071e14"}
Sep 10 23:48:16.686449 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:16.686409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d" event={"ID":"a21186bb-54f4-4c9a-baeb-30e0e21ea30d","Type":"ContainerStarted","Data":"2bd3fe2e5c5dd4cf84d9ec7d918c962ee04650f1590817650d9d39abc8fca35a"}
Sep 10 23:48:16.703419 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:16.703377    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d" podStartSLOduration=2.533271962 podStartE2EDuration="2.703360834s" totalImagesPullingTime="170.088872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:48:16.702760915 +0000 UTC m=+738.658992731" watchObservedRunningTime="2026-09-10 23:48:16.703360834 +0000 UTC m=+738.659592635"
Sep 10 23:48:17.188649 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:17.188610    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:48:25.264255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:25.264222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:25.264714 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:25.264267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:25.276692 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:25.276665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:48:25.736113 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:48:25.736034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f582d"
Sep 10 23:49:35.709393 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:35.709265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:39.479928 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:39.479894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:39.572497 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:39.572463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:40.553707 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:40.553670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:40.649918 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:40.649885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:45.831965 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:45.831916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:45.894441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:45.894410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:56.418586 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:56.418548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:56.544842 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:56.544809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:56.832218 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:56.832172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:57.380824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:57.380791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:58.048123 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.048084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:58.156155 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.156122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:58.498234 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.498156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:58.594893 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.594860    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7756-11f03a"]
Sep 10 23:49:58.598249 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.598228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:49:58.603109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.603083    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wtk84\""
Sep 10 23:49:58.604930 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.604910    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7756-11f03a"]
Sep 10 23:49:58.778347 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.778319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bfb\" (UniqueName: \"kubernetes.io/projected/47b8c257-d6a7-4bac-8543-a2e1f2c7af55-kube-api-access-62bfb\") pod \"test-curl-7756-11f03a\" (UID: \"47b8c257-d6a7-4bac-8543-a2e1f2c7af55\") " pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:49:58.879619 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.879584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bfb\" (UniqueName: \"kubernetes.io/projected/47b8c257-d6a7-4bac-8543-a2e1f2c7af55-kube-api-access-62bfb\") pod \"test-curl-7756-11f03a\" (UID: \"47b8c257-d6a7-4bac-8543-a2e1f2c7af55\") " pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:49:58.894646 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.894618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bfb\" (UniqueName: \"kubernetes.io/projected/47b8c257-d6a7-4bac-8543-a2e1f2c7af55-kube-api-access-62bfb\") pod \"test-curl-7756-11f03a\" (UID: \"47b8c257-d6a7-4bac-8543-a2e1f2c7af55\") " pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:49:58.907338 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:58.907316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:49:59.177344 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:59.177255    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7756-11f03a"]
Sep 10 23:49:59.180760 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:49:59.180725    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47b8c257_d6a7_4bac_8543_a2e1f2c7af55.slice/crio-a9114f6099aa8a9a55390483635d5484e9eb8e0418d4bbdb73595f91d02feb8c WatchSource:0}: Error finding container a9114f6099aa8a9a55390483635d5484e9eb8e0418d4bbdb73595f91d02feb8c: Status 404 returned error can't find the container with id a9114f6099aa8a9a55390483635d5484e9eb8e0418d4bbdb73595f91d02feb8c
Sep 10 23:49:59.409865 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:59.409832    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:49:59.866884 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:59.866822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:59.963647 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:49:59.963606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:00.147604 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:00.147523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7756-11f03a" event={"ID":"47b8c257-d6a7-4bac-8543-a2e1f2c7af55","Type":"ContainerStarted","Data":"a9114f6099aa8a9a55390483635d5484e9eb8e0418d4bbdb73595f91d02feb8c"}
Sep 10 23:50:01.068089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.068057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:01.153196 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.153161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7756-11f03a" event={"ID":"47b8c257-d6a7-4bac-8543-a2e1f2c7af55","Type":"ContainerStarted","Data":"054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6"}
Sep 10 23:50:01.174231 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.174184    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7756-11f03a" podStartSLOduration=1.369072987 podStartE2EDuration="3.174168809s" totalImagesPullingTime="1.805095822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:01.172928512 +0000 UTC m=+843.129160314" watchObservedRunningTime="2026-09-10 23:50:01.174168809 +0000 UTC m=+843.130400612"
Sep 10 23:50:01.174669 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.174641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:01.278031 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.277998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:01.696358 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.696308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:01.777322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.777266    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7756-11f03a"]
Sep 10 23:50:01.779595 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.779567    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7756-11f03a"]
Sep 10 23:50:01.783828 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.783785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:01.822960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.822883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"]
Sep 10 23:50:01.823823 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.823797    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b8c257-d6a7-4bac-8543-a2e1f2c7af55" containerName="test-curl-7756-11f03a"
Sep 10 23:50:01.823823 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.823824    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b8c257-d6a7-4bac-8543-a2e1f2c7af55" containerName="test-curl-7756-11f03a"
Sep 10 23:50:01.824092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.824078    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b8c257-d6a7-4bac-8543-a2e1f2c7af55" containerName="test-curl-7756-11f03a"
Sep 10 23:50:01.828466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.828442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:01.831820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.831796    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4c5699-dockercfg-n2dbs\""
Sep 10 23:50:01.832023 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.831829    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4c5699\""
Sep 10 23:50:01.839349 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.839322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"]
Sep 10 23:50:01.933615 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.933581    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7756-40798d"]
Sep 10 23:50:01.936929 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.936910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:01.959441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:01.959415    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7756-40798d"]
Sep 10 23:50:02.006908 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.006874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed46c29a-db00-4cbf-81f6-bd97cad198b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:02.007065 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.006935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznh4\" (UniqueName: \"kubernetes.io/projected/ed46c29a-db00-4cbf-81f6-bd97cad198b0-kube-api-access-wznh4\") pod \"payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:02.107458 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.107376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkp47\" (UniqueName: \"kubernetes.io/projected/4beb0140-f365-43cc-bd62-18deaa6030cc-kube-api-access-wkp47\") pod \"test-curl-7756-40798d\" (UID: \"4beb0140-f365-43cc-bd62-18deaa6030cc\") " pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:02.107458 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.107434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed46c29a-db00-4cbf-81f6-bd97cad198b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:02.107861 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.107522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznh4\" (UniqueName: \"kubernetes.io/projected/ed46c29a-db00-4cbf-81f6-bd97cad198b0-kube-api-access-wznh4\") pod \"payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:02.108061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.108044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed46c29a-db00-4cbf-81f6-bd97cad198b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:02.117070 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.117047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznh4\" (UniqueName: \"kubernetes.io/projected/ed46c29a-db00-4cbf-81f6-bd97cad198b0-kube-api-access-wznh4\") pod \"payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:02.141448 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.141423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:02.158706 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:02.158676    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7756-11f03a" container="test-curl-7756-11f03a"
Sep 10 23:50:02.209397 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.209364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkp47\" (UniqueName: \"kubernetes.io/projected/4beb0140-f365-43cc-bd62-18deaa6030cc-kube-api-access-wkp47\") pod \"test-curl-7756-40798d\" (UID: \"4beb0140-f365-43cc-bd62-18deaa6030cc\") " pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:02.221016 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.220990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkp47\" (UniqueName: \"kubernetes.io/projected/4beb0140-f365-43cc-bd62-18deaa6030cc-kube-api-access-wkp47\") pod \"test-curl-7756-40798d\" (UID: \"4beb0140-f365-43cc-bd62-18deaa6030cc\") " pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:02.245699 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.245673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:02.416715 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.416683    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"]
Sep 10 23:50:02.417441 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:02.417411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded46c29a_db00_4cbf_81f6_bd97cad198b0.slice/crio-66d0ca9518dbd3e7a7c2c187152b893688d8d7d9005046eb9bb0159f1839c8dc WatchSource:0}: Error finding container 66d0ca9518dbd3e7a7c2c187152b893688d8d7d9005046eb9bb0159f1839c8dc: Status 404 returned error can't find the container with id 66d0ca9518dbd3e7a7c2c187152b893688d8d7d9005046eb9bb0159f1839c8dc
Sep 10 23:50:02.524708 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:02.524686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7756-40798d"]
Sep 10 23:50:02.526894 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:02.526870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4beb0140_f365_43cc_bd62_18deaa6030cc.slice/crio-28d3967cc2c34308e6df24730ab93d52cc5477b200ac5c2ba17ce5985bb8b71c WatchSource:0}: Error finding container 28d3967cc2c34308e6df24730ab93d52cc5477b200ac5c2ba17ce5985bb8b71c: Status 404 returned error can't find the container with id 28d3967cc2c34308e6df24730ab93d52cc5477b200ac5c2ba17ce5985bb8b71c
Sep 10 23:50:03.168342 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:03.168299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7756-40798d" event={"ID":"4beb0140-f365-43cc-bd62-18deaa6030cc","Type":"ContainerStarted","Data":"d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8"}
Sep 10 23:50:03.168843 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:03.168350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7756-40798d" event={"ID":"4beb0140-f365-43cc-bd62-18deaa6030cc","Type":"ContainerStarted","Data":"28d3967cc2c34308e6df24730ab93d52cc5477b200ac5c2ba17ce5985bb8b71c"}
Sep 10 23:50:03.169848 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:03.169803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g" event={"ID":"ed46c29a-db00-4cbf-81f6-bd97cad198b0","Type":"ContainerStarted","Data":"66d0ca9518dbd3e7a7c2c187152b893688d8d7d9005046eb9bb0159f1839c8dc"}
Sep 10 23:50:03.169974 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:03.169890    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7756-11f03a" podUID="47b8c257-d6a7-4bac-8543-a2e1f2c7af55" containerName="test-curl-7756-11f03a" containerID="cri-o://054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6" gracePeriod=2
Sep 10 23:50:03.185686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:03.185628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7756-40798d" podStartSLOduration=2.185615359 podStartE2EDuration="2.185615359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:03.182685643 +0000 UTC m=+845.138917449" watchObservedRunningTime="2026-09-10 23:50:03.185615359 +0000 UTC m=+845.141847162"
Sep 10 23:50:03.850985 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:03.850952    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7756-40798d"]
Sep 10 23:50:03.858930 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:03.858856    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7756-40798d"]
Sep 10 23:50:04.175643 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:04.175570    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7756-40798d" container="test-curl-7756-40798d"
Sep 10 23:50:04.420166 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:04.420130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:04.735152 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:04.735118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:04.809816 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:04.809786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:04.914143 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:04.914110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:05.126543 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.126510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:05.182540 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.182473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g" event={"ID":"ed46c29a-db00-4cbf-81f6-bd97cad198b0","Type":"ContainerStarted","Data":"58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27"}
Sep 10 23:50:05.182540 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.182513    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7756-40798d" podUID="4beb0140-f365-43cc-bd62-18deaa6030cc" containerName="test-curl-7756-40798d" containerID="cri-o://d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8" gracePeriod=2
Sep 10 23:50:05.213822 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.213760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g" podStartSLOduration=1.843605593 podStartE2EDuration="4.213737347s" totalImagesPullingTime="2.370131754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:05.210994585 +0000 UTC m=+847.167226389" watchObservedRunningTime="2026-09-10 23:50:05.213737347 +0000 UTC m=+847.169969151"
Sep 10 23:50:05.230444 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.230406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:05.335775 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.335737    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:50:05.340772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.340743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62bfb\" (UniqueName: \"kubernetes.io/projected/47b8c257-d6a7-4bac-8543-a2e1f2c7af55-kube-api-access-62bfb\") pod \"47b8c257-d6a7-4bac-8543-a2e1f2c7af55\" (UID: \"47b8c257-d6a7-4bac-8543-a2e1f2c7af55\") "
Sep 10 23:50:05.343865 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.343521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b8c257-d6a7-4bac-8543-a2e1f2c7af55-kube-api-access-62bfb" (OuterVolumeSpecName: "kube-api-access-62bfb") pod "47b8c257-d6a7-4bac-8543-a2e1f2c7af55" (UID: "47b8c257-d6a7-4bac-8543-a2e1f2c7af55"). InnerVolumeSpecName "kube-api-access-62bfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:05.422023 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.421940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:05.442311 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.442261    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62bfb\" (UniqueName: \"kubernetes.io/projected/47b8c257-d6a7-4bac-8543-a2e1f2c7af55-kube-api-access-62bfb\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:50:05.590899 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:05.590859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:06.188713 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:06.188683    2578 generic.go:356] "Generic (PLEG): container finished" podID="47b8c257-d6a7-4bac-8543-a2e1f2c7af55" containerID="054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6" exitCode=137
Sep 10 23:50:06.189195 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:06.188735    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:50:06.189195 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:06.188786    2578 scope.go:117] "RemoveContainer" containerID="054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6"
Sep 10 23:50:06.189195 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:06.189085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:06.197130 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:06.197109    2578 scope.go:117] "RemoveContainer" containerID="054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6"
Sep 10 23:50:06.197418 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:06.197398    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6\": container with ID starting with 054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6 not found: ID does not exist" containerID="054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6"
Sep 10 23:50:06.197494 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:06.197429    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6"} err="failed to get container status \"054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6\": rpc error: code = NotFound desc = could not find container \"054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6\": container with ID starting with 054ca1927fd4df44dfc222b302b8ac02aebb92b23c8ecb3f828c94527f5c81f6 not found: ID does not exist"
Sep 10 23:50:06.202024 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:06.202002    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7756-11f03a\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="47b8c257-d6a7-4bac-8543-a2e1f2c7af55" pod="opendatahub/test-curl-7756-11f03a"
Sep 10 23:50:06.535891 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:06.535852    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b8c257-d6a7-4bac-8543-a2e1f2c7af55" path="/var/lib/kubelet/pods/47b8c257-d6a7-4bac-8543-a2e1f2c7af55/volumes"
Sep 10 23:50:07.370423 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:07.370394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:07.459462 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:07.459394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkp47\" (UniqueName: \"kubernetes.io/projected/4beb0140-f365-43cc-bd62-18deaa6030cc-kube-api-access-wkp47\") pod \"4beb0140-f365-43cc-bd62-18deaa6030cc\" (UID: \"4beb0140-f365-43cc-bd62-18deaa6030cc\") "
Sep 10 23:50:07.461384 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:07.461364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beb0140-f365-43cc-bd62-18deaa6030cc-kube-api-access-wkp47" (OuterVolumeSpecName: "kube-api-access-wkp47") pod "4beb0140-f365-43cc-bd62-18deaa6030cc" (UID: "4beb0140-f365-43cc-bd62-18deaa6030cc"). InnerVolumeSpecName "kube-api-access-wkp47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:07.560991 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:07.560965    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkp47\" (UniqueName: \"kubernetes.io/projected/4beb0140-f365-43cc-bd62-18deaa6030cc-kube-api-access-wkp47\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:50:07.845322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:07.845269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:07.952186 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:07.952150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:08.120798 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.120709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:08.200669 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.200633    2578 generic.go:356] "Generic (PLEG): container finished" podID="4beb0140-f365-43cc-bd62-18deaa6030cc" containerID="d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8" exitCode=137
Sep 10 23:50:08.200852 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.200674    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:08.200852 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.200684    2578 scope.go:117] "RemoveContainer" containerID="d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8"
Sep 10 23:50:08.207733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.207701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:08.209772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.209755    2578 scope.go:117] "RemoveContainer" containerID="d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8"
Sep 10 23:50:08.210030 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:08.210009    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8\": container with ID starting with d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8 not found: ID does not exist" containerID="d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8"
Sep 10 23:50:08.210092 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.210039    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8"} err="failed to get container status \"d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8\": rpc error: code = NotFound desc = could not find container \"d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8\": container with ID starting with d42c2af1e6e0eab7639e6103582fd83410d16529ed2230e4261dd5e1f4e910a8 not found: ID does not exist"
Sep 10 23:50:08.211106 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:08.211083    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7756-40798d\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="4beb0140-f365-43cc-bd62-18deaa6030cc" pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:08.532018 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:08.531980    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7756-40798d\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="4beb0140-f365-43cc-bd62-18deaa6030cc" pod="opendatahub/test-curl-7756-40798d"
Sep 10 23:50:08.533712 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.533681    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4beb0140-f365-43cc-bd62-18deaa6030cc" path="/var/lib/kubelet/pods/4beb0140-f365-43cc-bd62-18deaa6030cc/volumes"
Sep 10 23:50:08.989566 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:08.989475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:09.735586 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.735556    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"]
Sep 10 23:50:09.736051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.735921    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4beb0140-f365-43cc-bd62-18deaa6030cc" containerName="test-curl-7756-40798d"
Sep 10 23:50:09.736051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.735937    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4beb0140-f365-43cc-bd62-18deaa6030cc" containerName="test-curl-7756-40798d"
Sep 10 23:50:09.736051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.736013    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4beb0140-f365-43cc-bd62-18deaa6030cc" containerName="test-curl-7756-40798d"
Sep 10 23:50:09.740300 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.740264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.743262 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.743239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-b3de77-serving-cert\""
Sep 10 23:50:09.744051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.744025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:50:09.744158 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.744060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-78nn9\""
Sep 10 23:50:09.744378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.744358    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:50:09.744453 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.744384    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:50:09.753235 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.753205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"]
Sep 10 23:50:09.778667 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.778637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl6dx\" (UniqueName: \"kubernetes.io/projected/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-kube-api-access-nl6dx\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.778769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.778676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-openshift-service-ca\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.778769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.778747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.778867 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.778782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-metrics-certs\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.788459 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.788430    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:09.879595 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.879566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-metrics-certs\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.879724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.879630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl6dx\" (UniqueName: \"kubernetes.io/projected/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-kube-api-access-nl6dx\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.879724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.879665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-openshift-service-ca\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.879824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.879802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.879964 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:09.879945    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-serving-cert: secret "maas-api-e2e-autores-b3de77-serving-cert" not found
Sep 10 23:50:09.880034 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:09.880023    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls podName:46d1829b-1ac4-4c46-8da0-44c2a1bdd71a nodeName:}" failed. No retries permitted until 2026-09-10 23:50:10.380000801 +0000 UTC m=+852.336232597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls") pod "maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld" (UID: "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a") : secret "maas-api-e2e-autores-b3de77-serving-cert" not found
Sep 10 23:50:09.880350 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.880333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-openshift-service-ca\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.882157 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.882137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-metrics-certs\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:09.888660 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.888601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:09.894331 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:09.894311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl6dx\" (UniqueName: \"kubernetes.io/projected/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-kube-api-access-nl6dx\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:10.109839 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.109807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:10.385263 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.385186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:10.386374 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.386350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:10.387635 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.387616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls\") pod \"maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:10.491803 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.491771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:10.655353 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.652167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:10.778466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.778431    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"]
Sep 10 23:50:10.783673 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.783648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.788605 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.788580    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6c506b66-serving-cert\""
Sep 10 23:50:10.802643 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.802621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"]
Sep 10 23:50:10.825706 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.825675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:10.888179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.888142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.888396 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.888207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817373d-6ae5-4f05-a043-08d3957a1fb8-openshift-service-ca\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.888396 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.888259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-metrics-certs\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.888396 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.888341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58tdf\" (UniqueName: \"kubernetes.io/projected/9817373d-6ae5-4f05-a043-08d3957a1fb8-kube-api-access-58tdf\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.903736 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.903690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:10.937533 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.937510    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"]
Sep 10 23:50:10.939222 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:10.939197    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d1829b_1ac4_4c46_8da0_44c2a1bdd71a.slice/crio-e1e9ed556d823eee6984dba3b04c56b6fa9852a525402e8e8744efb3ce8f12e3 WatchSource:0}: Error finding container e1e9ed556d823eee6984dba3b04c56b6fa9852a525402e8e8744efb3ce8f12e3: Status 404 returned error can't find the container with id e1e9ed556d823eee6984dba3b04c56b6fa9852a525402e8e8744efb3ce8f12e3
Sep 10 23:50:10.989097 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.989075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817373d-6ae5-4f05-a043-08d3957a1fb8-openshift-service-ca\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.989207 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.989115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-metrics-certs\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.989207 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.989157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58tdf\" (UniqueName: \"kubernetes.io/projected/9817373d-6ae5-4f05-a043-08d3957a1fb8-kube-api-access-58tdf\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.989339 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.989222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.989398 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:10.989388    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-serving-cert: secret "maas-api-e2e-ait-6c506b66-serving-cert" not found
Sep 10 23:50:10.989466 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:10.989454    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls podName:9817373d-6ae5-4f05-a043-08d3957a1fb8 nodeName:}" failed. No retries permitted until 2026-09-10 23:50:11.489433067 +0000 UTC m=+853.445664849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls") pod "maas-api-e2e-ait-6c506b66-b54b7597d-nfp74" (UID: "9817373d-6ae5-4f05-a043-08d3957a1fb8") : secret "maas-api-e2e-ait-6c506b66-serving-cert" not found
Sep 10 23:50:10.989774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.989751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817373d-6ae5-4f05-a043-08d3957a1fb8-openshift-service-ca\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.991493 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.991474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-metrics-certs\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:10.998428 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:10.998408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58tdf\" (UniqueName: \"kubernetes.io/projected/9817373d-6ae5-4f05-a043-08d3957a1fb8-kube-api-access-58tdf\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:11.126587 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.126557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:11.216955 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.216882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld" event={"ID":"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a","Type":"ContainerStarted","Data":"e1e9ed556d823eee6984dba3b04c56b6fa9852a525402e8e8744efb3ce8f12e3"}
Sep 10 23:50:11.309402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.309356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:11.413154 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.413002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:11.494589 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.494384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:11.497598 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.497565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls\") pod \"maas-api-e2e-ait-6c506b66-b54b7597d-nfp74\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:11.697131 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.697096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:11.865541 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.865493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:11.985848 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:11.985256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"]
Sep 10 23:50:12.807468 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:12.807427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:13.050203 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:13.050164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9817373d_6ae5_4f05_a043_08d3957a1fb8.slice/crio-22aaae906119e79f84f09c9a28f0c6c16c42ea3cce74c954a77b52d635acfda7 WatchSource:0}: Error finding container 22aaae906119e79f84f09c9a28f0c6c16c42ea3cce74c954a77b52d635acfda7: Status 404 returned error can't find the container with id 22aaae906119e79f84f09c9a28f0c6c16c42ea3cce74c954a77b52d635acfda7
Sep 10 23:50:13.235056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:13.235015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld" event={"ID":"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a","Type":"ContainerStarted","Data":"66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab"}
Sep 10 23:50:13.235482 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:13.235458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:13.236580 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:13.236538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74" event={"ID":"9817373d-6ae5-4f05-a043-08d3957a1fb8","Type":"ContainerStarted","Data":"22aaae906119e79f84f09c9a28f0c6c16c42ea3cce74c954a77b52d635acfda7"}
Sep 10 23:50:13.253512 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:13.253454    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld" podStartSLOduration=2.092478037 podStartE2EDuration="4.253436251s" totalImagesPullingTime="2.160958214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:13.252608074 +0000 UTC m=+855.208839872" watchObservedRunningTime="2026-09-10 23:50:13.253436251 +0000 UTC m=+855.209668058"
Sep 10 23:50:13.507062 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:13.507035    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:50:13.575267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:13.575236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:14.243454 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:14.243419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74" event={"ID":"9817373d-6ae5-4f05-a043-08d3957a1fb8","Type":"ContainerStarted","Data":"c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6"}
Sep 10 23:50:14.243907 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:14.243518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:14.266070 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:14.266012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74" podStartSLOduration=3.86106071 podStartE2EDuration="4.265995643s" totalImagesPullingTime="404.934933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:14.265046345 +0000 UTC m=+856.221278147" watchObservedRunningTime="2026-09-10 23:50:14.265995643 +0000 UTC m=+856.222227445"
Sep 10 23:50:14.485166 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:14.485130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:14.787504 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:14.787466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:14.892249 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:14.892216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:15.088928 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:15.088841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:15.199515 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:15.199475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:15.356014 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:15.355896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:15.462724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:15.462689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:15.563505 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:15.563471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:16.366569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:16.366535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:17.196798 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:17.196767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:50:17.213872 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:17.213836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:17.823651 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:17.823614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:17.934263 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:17.934223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:18.035631 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:18.035602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:18.309940 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:18.309905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:18.407833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:18.407799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:19.369606 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:19.369569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:19.880798 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:19.880763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:20.199481 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:20.199398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:20.256194 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:20.256158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:20.256371 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:20.256236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:50:21.056688 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.056651    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"]
Sep 10 23:50:21.060267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.060246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.062982 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.062958    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-948e64-serving-cert\""
Sep 10 23:50:21.073292 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.073240    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"]
Sep 10 23:50:21.160402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.160369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:21.187579 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.187541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.187709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.187596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-metrics-certs\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.187709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.187645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9c2w\" (UniqueName: \"kubernetes.io/projected/33cb34c5-8674-4568-bf00-238a5b5b9059-kube-api-access-n9c2w\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.187791 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.187732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cb34c5-8674-4568-bf00-238a5b5b9059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.262735 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.262707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:21.288485 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.288456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9c2w\" (UniqueName: \"kubernetes.io/projected/33cb34c5-8674-4568-bf00-238a5b5b9059-kube-api-access-n9c2w\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.288608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.288513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cb34c5-8674-4568-bf00-238a5b5b9059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.288608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.288552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.288608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.288583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-metrics-certs\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.288776 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:21.288690    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-serving-cert: secret "maas-api-e2e-inf-b-948e64-serving-cert" not found
Sep 10 23:50:21.288776 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:21.288750    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls podName:33cb34c5-8674-4568-bf00-238a5b5b9059 nodeName:}" failed. No retries permitted until 2026-09-10 23:50:21.788734503 +0000 UTC m=+863.744966300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls") pod "maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc" (UID: "33cb34c5-8674-4568-bf00-238a5b5b9059") : secret "maas-api-e2e-inf-b-948e64-serving-cert" not found
Sep 10 23:50:21.289234 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.289215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cb34c5-8674-4568-bf00-238a5b5b9059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.290904 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.290884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-metrics-certs\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.301096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.301070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9c2w\" (UniqueName: \"kubernetes.io/projected/33cb34c5-8674-4568-bf00-238a5b5b9059-kube-api-access-n9c2w\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.427370 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.427299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:21.793480 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.793448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.795783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.795753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls\") pod \"maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:21.938873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.938824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:21.970856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:21.970827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:22.009531 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.009497    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"]
Sep 10 23:50:22.013773 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.013756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.017488 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.017139    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-948e64\""
Sep 10 23:50:22.020800 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.020776    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"]
Sep 10 23:50:22.036928 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.028968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.052819 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.037550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:22.052819 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.038112    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-948e64-dockercfg-7xc8w\""
Sep 10 23:50:22.054452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.054353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"]
Sep 10 23:50:22.058761 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.058739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"]
Sep 10 23:50:22.095546 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.095523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndkql\" (UniqueName: \"kubernetes.io/projected/555d01b6-5141-4a91-a144-804689cca34d-kube-api-access-ndkql\") pod \"payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.095649 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.095559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr279\" (UniqueName: \"kubernetes.io/projected/5753bc76-f54b-4212-9964-aeb2302fa8b3-kube-api-access-kr279\") pod \"payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.095649 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.095612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5753bc76-f54b-4212-9964-aeb2302fa8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.095737 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.095649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555d01b6-5141-4a91-a144-804689cca34d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.196749 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.196721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndkql\" (UniqueName: \"kubernetes.io/projected/555d01b6-5141-4a91-a144-804689cca34d-kube-api-access-ndkql\") pod \"payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.196872 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.196763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr279\" (UniqueName: \"kubernetes.io/projected/5753bc76-f54b-4212-9964-aeb2302fa8b3-kube-api-access-kr279\") pod \"payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.196872 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.196800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5753bc76-f54b-4212-9964-aeb2302fa8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.196872 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.196831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555d01b6-5141-4a91-a144-804689cca34d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.197730 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.197708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555d01b6-5141-4a91-a144-804689cca34d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.197730 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.197722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5753bc76-f54b-4212-9964-aeb2302fa8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.205521 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.205498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr279\" (UniqueName: \"kubernetes.io/projected/5753bc76-f54b-4212-9964-aeb2302fa8b3-kube-api-access-kr279\") pod \"payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.205700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.205669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndkql\" (UniqueName: \"kubernetes.io/projected/555d01b6-5141-4a91-a144-804689cca34d-kube-api-access-ndkql\") pod \"payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.251544 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.251509    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"]
Sep 10 23:50:22.253654 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:22.253628    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cb34c5_8674_4568_bf00_238a5b5b9059.slice/crio-ed2bc45fe530feae466355f1d15dd70371593ae723400c60386e17b2f3a06844 WatchSource:0}: Error finding container ed2bc45fe530feae466355f1d15dd70371593ae723400c60386e17b2f3a06844: Status 404 returned error can't find the container with id ed2bc45fe530feae466355f1d15dd70371593ae723400c60386e17b2f3a06844
Sep 10 23:50:22.288031 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.288003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc" event={"ID":"33cb34c5-8674-4568-bf00-238a5b5b9059","Type":"ContainerStarted","Data":"ed2bc45fe530feae466355f1d15dd70371593ae723400c60386e17b2f3a06844"}
Sep 10 23:50:22.323096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.323031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:22.339698 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.339659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:22.359048 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.359026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:22.637743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.637665    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"]
Sep 10 23:50:22.641943 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:22.641910    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555d01b6_5141_4a91_a144_804689cca34d.slice/crio-2328768a328eaaf02efa5c7217c39fb5d307d42ce371880cecb64782881a3b7e WatchSource:0}: Error finding container 2328768a328eaaf02efa5c7217c39fb5d307d42ce371880cecb64782881a3b7e: Status 404 returned error can't find the container with id 2328768a328eaaf02efa5c7217c39fb5d307d42ce371880cecb64782881a3b7e
Sep 10 23:50:22.649038 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:22.649018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"]
Sep 10 23:50:22.650362 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:22.650337    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5753bc76_f54b_4212_9964_aeb2302fa8b3.slice/crio-845cee4ad4685f574081e63e7d059b8d1d78b7d10c407c545a8d89881578d3f9 WatchSource:0}: Error finding container 845cee4ad4685f574081e63e7d059b8d1d78b7d10c407c545a8d89881578d3f9: Status 404 returned error can't find the container with id 845cee4ad4685f574081e63e7d059b8d1d78b7d10c407c545a8d89881578d3f9
Sep 10 23:50:23.295487 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.295453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc" event={"ID":"33cb34c5-8674-4568-bf00-238a5b5b9059","Type":"ContainerStarted","Data":"8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec"}
Sep 10 23:50:23.295916 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.295541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:23.297425 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.297387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b" event={"ID":"5753bc76-f54b-4212-9964-aeb2302fa8b3","Type":"ContainerStarted","Data":"9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16"}
Sep 10 23:50:23.297565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.297454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b" event={"ID":"5753bc76-f54b-4212-9964-aeb2302fa8b3","Type":"ContainerStarted","Data":"845cee4ad4685f574081e63e7d059b8d1d78b7d10c407c545a8d89881578d3f9"}
Sep 10 23:50:23.297565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.297487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:23.299068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.299046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp" event={"ID":"555d01b6-5141-4a91-a144-804689cca34d","Type":"ContainerStarted","Data":"1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5"}
Sep 10 23:50:23.299203 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.299080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp" event={"ID":"555d01b6-5141-4a91-a144-804689cca34d","Type":"ContainerStarted","Data":"2328768a328eaaf02efa5c7217c39fb5d307d42ce371880cecb64782881a3b7e"}
Sep 10 23:50:23.299203 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.299099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:23.304701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.304676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:23.317261 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.317220    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc" podStartSLOduration=1.9270840169999999 podStartE2EDuration="2.3172064s" totalImagesPullingTime="390.122383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:23.315387613 +0000 UTC m=+865.271619440" watchObservedRunningTime="2026-09-10 23:50:23.3172064 +0000 UTC m=+865.273438202"
Sep 10 23:50:23.337286 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.337230    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp" podStartSLOduration=2.33721265 podStartE2EDuration="2.33721265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:23.336708268 +0000 UTC m=+865.292940074" watchObservedRunningTime="2026-09-10 23:50:23.33721265 +0000 UTC m=+865.293444455"
Sep 10 23:50:23.354802 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:23.354765    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b" podStartSLOduration=2.354753187 podStartE2EDuration="2.354753187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:23.353371747 +0000 UTC m=+865.309603550" watchObservedRunningTime="2026-09-10 23:50:23.354753187 +0000 UTC m=+865.310984990"
Sep 10 23:50:24.151547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:24.151504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:25.094650 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:25.094607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:25.274849 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:25.274816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:25.379637 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:25.379544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:25.489676 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:25.489641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:25.636034 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:25.635946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:25.983810 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:25.983732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:27.036705 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:27.036672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:27.535162 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:27.535125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:28.092890 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:28.092853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:28.852234 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:28.852186    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:29.309746 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:29.309718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:50:29.694290 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:29.694205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:50:29.791408 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:29.791370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:30.732139 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:30.732105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:31.533239 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:31.533196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:32.424151 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:32.424115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:32.524167 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:32.524125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:32.624161 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:32.624114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:33.281959 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:33.281926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:33.954280 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:33.954238    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"]
Sep 10 23:50:33.957975 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:33.957959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:33.960567 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:33.960548    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-ca420b8d-kserve-self-signed-certs\""
Sep 10 23:50:33.967750 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:33.967725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"]
Sep 10 23:50:34.109172 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.109137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tmp-dir\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.109366 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.109188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-dshm\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.109366 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.109248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdnn\" (UniqueName: \"kubernetes.io/projected/a24d016f-15ec-4aed-a8c3-dfe85091cc94-kube-api-access-4bdnn\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.109366 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.109297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-home\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.109366 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.109340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-model-cache\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.109366 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.109360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tls-certs\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.210546 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-home\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.210546 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-model-cache\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.210756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tls-certs\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.210756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tmp-dir\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.210756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-dshm\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.210910 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdnn\" (UniqueName: \"kubernetes.io/projected/a24d016f-15ec-4aed-a8c3-dfe85091cc94-kube-api-access-4bdnn\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.210910 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-model-cache\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.211009 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-home\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.211009 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.210996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tmp-dir\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.212848 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.212824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-dshm\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.213055 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.213038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tls-certs\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.218825 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.218799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdnn\" (UniqueName: \"kubernetes.io/projected/a24d016f-15ec-4aed-a8c3-dfe85091cc94-kube-api-access-4bdnn\") pod \"e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") " pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.267293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.267255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:34.281067 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.281043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:34.306511 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.306467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:50:34.306706 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.306602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:50:34.379192 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.379150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:34.541071 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.541001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"]
Sep 10 23:50:34.544144 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:34.544115    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda24d016f_15ec_4aed_a8c3_dfe85091cc94.slice/crio-4b91e49887c1b240cc1d87264f9a34e324d389463717af8c6c7dc15b38af6156 WatchSource:0}: Error finding container 4b91e49887c1b240cc1d87264f9a34e324d389463717af8c6c7dc15b38af6156: Status 404 returned error can't find the container with id 4b91e49887c1b240cc1d87264f9a34e324d389463717af8c6c7dc15b38af6156
Sep 10 23:50:34.979775 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:34.979692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:35.364048 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:35.364013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr" event={"ID":"a24d016f-15ec-4aed-a8c3-dfe85091cc94","Type":"ContainerStarted","Data":"00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445"}
Sep 10 23:50:35.364048 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:35.364051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr" event={"ID":"a24d016f-15ec-4aed-a8c3-dfe85091cc94","Type":"ContainerStarted","Data":"4b91e49887c1b240cc1d87264f9a34e324d389463717af8c6c7dc15b38af6156"}
Sep 10 23:50:35.381952 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:35.381908    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr" podStartSLOduration=2.3818933380000002 podStartE2EDuration="2.381893338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:35.380692876 +0000 UTC m=+877.336924682" watchObservedRunningTime="2026-09-10 23:50:35.381893338 +0000 UTC m=+877.338125165"
Sep 10 23:50:35.785293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:35.781586    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:36.632379 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.632148    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"]
Sep 10 23:50:36.632929 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.632442    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74" podUID="9817373d-6ae5-4f05-a043-08d3957a1fb8" containerName="maas-api" containerID="cri-o://c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6" gracePeriod=30
Sep 10 23:50:36.660013 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.659982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:36.737119 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.737081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:36.800792 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.800648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:36.883979 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.883906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:36.887188 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.887165    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:36.952089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:36.952032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:37.037327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.037295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58tdf\" (UniqueName: \"kubernetes.io/projected/9817373d-6ae5-4f05-a043-08d3957a1fb8-kube-api-access-58tdf\") pod \"9817373d-6ae5-4f05-a043-08d3957a1fb8\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") "
Sep 10 23:50:37.037511 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.037354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls\") pod \"9817373d-6ae5-4f05-a043-08d3957a1fb8\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") "
Sep 10 23:50:37.037511 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.037396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817373d-6ae5-4f05-a043-08d3957a1fb8-openshift-service-ca\") pod \"9817373d-6ae5-4f05-a043-08d3957a1fb8\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") "
Sep 10 23:50:37.037511 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.037487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-metrics-certs\") pod \"9817373d-6ae5-4f05-a043-08d3957a1fb8\" (UID: \"9817373d-6ae5-4f05-a043-08d3957a1fb8\") "
Sep 10 23:50:37.038086 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.038050    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9817373d-6ae5-4f05-a043-08d3957a1fb8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9817373d-6ae5-4f05-a043-08d3957a1fb8" (UID: "9817373d-6ae5-4f05-a043-08d3957a1fb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:37.039908 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.039885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9817373d-6ae5-4f05-a043-08d3957a1fb8" (UID: "9817373d-6ae5-4f05-a043-08d3957a1fb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:50:37.040044 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.040022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9817373d-6ae5-4f05-a043-08d3957a1fb8" (UID: "9817373d-6ae5-4f05-a043-08d3957a1fb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:50:37.040044 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.040036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9817373d-6ae5-4f05-a043-08d3957a1fb8-kube-api-access-58tdf" (OuterVolumeSpecName: "kube-api-access-58tdf") pod "9817373d-6ae5-4f05-a043-08d3957a1fb8" (UID: "9817373d-6ae5-4f05-a043-08d3957a1fb8"). InnerVolumeSpecName "kube-api-access-58tdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:37.056484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.056392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:37.138911 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.138826    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58tdf\" (UniqueName: \"kubernetes.io/projected/9817373d-6ae5-4f05-a043-08d3957a1fb8-kube-api-access-58tdf\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:50:37.138911 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.138855    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-maas-api-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:50:37.138911 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.138868    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817373d-6ae5-4f05-a043-08d3957a1fb8-openshift-service-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:50:37.138911 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.138877    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817373d-6ae5-4f05-a043-08d3957a1fb8-metrics-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:50:37.374547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.374506    2578 generic.go:356] "Generic (PLEG): container finished" podID="9817373d-6ae5-4f05-a043-08d3957a1fb8" containerID="c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6" exitCode=0
Sep 10 23:50:37.374705 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.374546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74" event={"ID":"9817373d-6ae5-4f05-a043-08d3957a1fb8","Type":"ContainerDied","Data":"c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6"}
Sep 10 23:50:37.374705 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.374611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74" event={"ID":"9817373d-6ae5-4f05-a043-08d3957a1fb8","Type":"ContainerDied","Data":"22aaae906119e79f84f09c9a28f0c6c16c42ea3cce74c954a77b52d635acfda7"}
Sep 10 23:50:37.374705 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.374559    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"
Sep 10 23:50:37.374868 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.374641    2578 scope.go:117] "RemoveContainer" containerID="c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6"
Sep 10 23:50:37.384534 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.384517    2578 scope.go:117] "RemoveContainer" containerID="c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6"
Sep 10 23:50:37.384795 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:50:37.384774    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6\": container with ID starting with c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6 not found: ID does not exist" containerID="c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6"
Sep 10 23:50:37.384902 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.384799    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6"} err="failed to get container status \"c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6\": rpc error: code = NotFound desc = could not find container \"c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6\": container with ID starting with c006d37354d8c1a480691ad53d65911c1cb09e53ec18bb746e8d2d8cce868ec6 not found: ID does not exist"
Sep 10 23:50:37.389929 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.389876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:37.397587 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.397565    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"]
Sep 10 23:50:37.404324 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.404305    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c506b66-b54b7597d-nfp74"]
Sep 10 23:50:37.546121 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.546089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:37.638448 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.638411    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:37.762468 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:37.762436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:38.084312 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:38.084216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:38.280239 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:38.280196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:38.393924 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:38.393836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:38.534645 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:38.534611    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9817373d-6ae5-4f05-a043-08d3957a1fb8" path="/var/lib/kubelet/pods/9817373d-6ae5-4f05-a043-08d3957a1fb8/volumes"
Sep 10 23:50:39.171558 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:39.171523    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:39.975032 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:39.974994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:40.524455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:40.524418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:40.631065 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:40.631017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:40.723944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:40.723911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:40.835195 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:40.835111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:41.127222 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.127143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:41.301758 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.301718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:41.404342 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.404233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:41.412008 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.411983    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"]
Sep 10 23:50:41.412401 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.412388    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9817373d-6ae5-4f05-a043-08d3957a1fb8" containerName="maas-api"
Sep 10 23:50:41.412447 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.412403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9817373d-6ae5-4f05-a043-08d3957a1fb8" containerName="maas-api"
Sep 10 23:50:41.412486 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.412481    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9817373d-6ae5-4f05-a043-08d3957a1fb8" containerName="maas-api"
Sep 10 23:50:41.416929 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.416911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.419556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.419537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-113befcd-dockercfg-cj2fk\""
Sep 10 23:50:41.419556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.419546    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-113befcd\""
Sep 10 23:50:41.425582 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.425551    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"]
Sep 10 23:50:41.444598 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.444574    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"]
Sep 10 23:50:41.447873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.447857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:41.457799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.457779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"]
Sep 10 23:50:41.581396 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.581363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gts4\" (UniqueName: \"kubernetes.io/projected/321f9c92-f813-452c-92de-0e5cbf57e62d-kube-api-access-6gts4\") pod \"payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") " pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:41.581783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.581433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586tf\" (UniqueName: \"kubernetes.io/projected/eb8b3540-eb80-42dc-b9a5-e0705554b9db-kube-api-access-586tf\") pod \"payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.581783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.581490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321f9c92-f813-452c-92de-0e5cbf57e62d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") " pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:41.581783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.581516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8b3540-eb80-42dc-b9a5-e0705554b9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.683079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.682982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586tf\" (UniqueName: \"kubernetes.io/projected/eb8b3540-eb80-42dc-b9a5-e0705554b9db-kube-api-access-586tf\") pod \"payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.683079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.683039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321f9c92-f813-452c-92de-0e5cbf57e62d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") " pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:41.683079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.683059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8b3540-eb80-42dc-b9a5-e0705554b9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.683401 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.683214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gts4\" (UniqueName: \"kubernetes.io/projected/321f9c92-f813-452c-92de-0e5cbf57e62d-kube-api-access-6gts4\") pod \"payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") " pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:41.683797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.683779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321f9c92-f813-452c-92de-0e5cbf57e62d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") " pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:41.683797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.683789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8b3540-eb80-42dc-b9a5-e0705554b9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.701479 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.701453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-586tf\" (UniqueName: \"kubernetes.io/projected/eb8b3540-eb80-42dc-b9a5-e0705554b9db-kube-api-access-586tf\") pod \"payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.701570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.701519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gts4\" (UniqueName: \"kubernetes.io/projected/321f9c92-f813-452c-92de-0e5cbf57e62d-kube-api-access-6gts4\") pod \"payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") " pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:41.729052 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.729025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:41.758617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:41.758587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:42.041581 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.041554    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"]
Sep 10 23:50:42.043478 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:42.043446    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8b3540_eb80_42dc_b9a5_e0705554b9db.slice/crio-551e78614e20a0bd160474dedf4919562aa255e05d71be3e97b7d8eb4e6dd1b9 WatchSource:0}: Error finding container 551e78614e20a0bd160474dedf4919562aa255e05d71be3e97b7d8eb4e6dd1b9: Status 404 returned error can't find the container with id 551e78614e20a0bd160474dedf4919562aa255e05d71be3e97b7d8eb4e6dd1b9
Sep 10 23:50:42.080901 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.080868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"]
Sep 10 23:50:42.083772 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:50:42.083746    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321f9c92_f813_452c_92de_0e5cbf57e62d.slice/crio-d4cd5e09b0d921004d5bb5ff0a3e7f5f567fc52cf3fe32d26f1189d17cf056b2 WatchSource:0}: Error finding container d4cd5e09b0d921004d5bb5ff0a3e7f5f567fc52cf3fe32d26f1189d17cf056b2: Status 404 returned error can't find the container with id d4cd5e09b0d921004d5bb5ff0a3e7f5f567fc52cf3fe32d26f1189d17cf056b2
Sep 10 23:50:42.134492 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.134461    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:50:42.155211 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.155185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:42.401329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.401216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d" event={"ID":"321f9c92-f813-452c-92de-0e5cbf57e62d","Type":"ContainerStarted","Data":"15f1825e8a3c0303d6dad01c5a90bc588b7fa7aeab9aac8c0e55bde6cc1df05a"}
Sep 10 23:50:42.401329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.401294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d" event={"ID":"321f9c92-f813-452c-92de-0e5cbf57e62d","Type":"ContainerStarted","Data":"d4cd5e09b0d921004d5bb5ff0a3e7f5f567fc52cf3fe32d26f1189d17cf056b2"}
Sep 10 23:50:42.401329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.401323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:42.402927 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.402895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb" event={"ID":"eb8b3540-eb80-42dc-b9a5-e0705554b9db","Type":"ContainerStarted","Data":"add032a1cfd766849213a302098a3e0400c31b67f4f07dde54a2a2dceb0da945"}
Sep 10 23:50:42.403054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.402935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb" event={"ID":"eb8b3540-eb80-42dc-b9a5-e0705554b9db","Type":"ContainerStarted","Data":"551e78614e20a0bd160474dedf4919562aa255e05d71be3e97b7d8eb4e6dd1b9"}
Sep 10 23:50:42.403054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.402965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:42.423350 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.423296    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d" podStartSLOduration=1.423264683 podStartE2EDuration="1.423264683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:42.421597918 +0000 UTC m=+884.377829722" watchObservedRunningTime="2026-09-10 23:50:42.423264683 +0000 UTC m=+884.379496488"
Sep 10 23:50:42.447908 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:42.447856    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb" podStartSLOduration=1.447838692 podStartE2EDuration="1.447838692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:42.446854768 +0000 UTC m=+884.403086583" watchObservedRunningTime="2026-09-10 23:50:42.447838692 +0000 UTC m=+884.404070494"
Sep 10 23:50:43.278317 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:43.278257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:44.090585 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:44.090548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:44.268187 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:44.268155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:44.268497 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:44.268467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:44.280678 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:44.280650    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:44.424691 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:44.424616    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:50:45.015250 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:45.015206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:45.749068 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:45.749031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:53.410899 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:53.410864    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:50:53.411321 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:53.411093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:50:56.098043 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:56.098004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:50:58.543114 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:58.543084    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:50:58.543574 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:50:58.543476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:51:01.862226 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:01.862182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:03.276621 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:03.276583    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:03.959312 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:03.959261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:04.749754 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:04.749717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:05.553369 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:05.553336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:06.309927 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:06.309894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:07.169115 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:07.169077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:08.035867 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:08.035828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:08.693355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:08.693308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:09.580810 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:09.580769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:11.084776 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:11.084743    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:51:11.767892 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:11.767850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:12.380454 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.380415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:12.404085 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.404054    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"]
Sep 10 23:51:12.404392 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.404364    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d" podUID="321f9c92-f813-452c-92de-0e5cbf57e62d" containerName="payload-processing" containerID="cri-o://15f1825e8a3c0303d6dad01c5a90bc588b7fa7aeab9aac8c0e55bde6cc1df05a" gracePeriod=30
Sep 10 23:51:12.414420 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.414393    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"]
Sep 10 23:51:12.414678 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.414654    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb" podUID="eb8b3540-eb80-42dc-b9a5-e0705554b9db" containerName="payload-pre-processing" containerID="cri-o://add032a1cfd766849213a302098a3e0400c31b67f4f07dde54a2a2dceb0da945" gracePeriod=30
Sep 10 23:51:12.480929 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.480896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:12.564992 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.564960    2578 generic.go:356] "Generic (PLEG): container finished" podID="321f9c92-f813-452c-92de-0e5cbf57e62d" containerID="15f1825e8a3c0303d6dad01c5a90bc588b7fa7aeab9aac8c0e55bde6cc1df05a" exitCode=0
Sep 10 23:51:12.565182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.565039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d" event={"ID":"321f9c92-f813-452c-92de-0e5cbf57e62d","Type":"ContainerDied","Data":"15f1825e8a3c0303d6dad01c5a90bc588b7fa7aeab9aac8c0e55bde6cc1df05a"}
Sep 10 23:51:12.566820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.566798    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb8b3540-eb80-42dc-b9a5-e0705554b9db" containerID="add032a1cfd766849213a302098a3e0400c31b67f4f07dde54a2a2dceb0da945" exitCode=0
Sep 10 23:51:12.566927 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.566873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb" event={"ID":"eb8b3540-eb80-42dc-b9a5-e0705554b9db","Type":"ContainerDied","Data":"add032a1cfd766849213a302098a3e0400c31b67f4f07dde54a2a2dceb0da945"}
Sep 10 23:51:12.582322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.582292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:12.651394 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.651316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:12.692961 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.692938    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:51:12.696116 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.696094    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:51:12.760744 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.760707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321f9c92-f813-452c-92de-0e5cbf57e62d-plugins-config-volume\") pod \"321f9c92-f813-452c-92de-0e5cbf57e62d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") "
Sep 10 23:51:12.760924 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.760781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8b3540-eb80-42dc-b9a5-e0705554b9db-plugins-config-volume\") pod \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") "
Sep 10 23:51:12.760924 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.760908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321f9c92-f813-452c-92de-0e5cbf57e62d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "321f9c92-f813-452c-92de-0e5cbf57e62d" (UID: "321f9c92-f813-452c-92de-0e5cbf57e62d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:12.761184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.761161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8b3540-eb80-42dc-b9a5-e0705554b9db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb8b3540-eb80-42dc-b9a5-e0705554b9db" (UID: "eb8b3540-eb80-42dc-b9a5-e0705554b9db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:12.761291 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.761237    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-586tf\" (UniqueName: \"kubernetes.io/projected/eb8b3540-eb80-42dc-b9a5-e0705554b9db-kube-api-access-586tf\") pod \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\" (UID: \"eb8b3540-eb80-42dc-b9a5-e0705554b9db\") "
Sep 10 23:51:12.761366 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.761294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gts4\" (UniqueName: \"kubernetes.io/projected/321f9c92-f813-452c-92de-0e5cbf57e62d-kube-api-access-6gts4\") pod \"321f9c92-f813-452c-92de-0e5cbf57e62d\" (UID: \"321f9c92-f813-452c-92de-0e5cbf57e62d\") "
Sep 10 23:51:12.761739 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.761717    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321f9c92-f813-452c-92de-0e5cbf57e62d-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:12.761831 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.761743    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8b3540-eb80-42dc-b9a5-e0705554b9db-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:12.764176 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.764152    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8b3540-eb80-42dc-b9a5-e0705554b9db-kube-api-access-586tf" (OuterVolumeSpecName: "kube-api-access-586tf") pod "eb8b3540-eb80-42dc-b9a5-e0705554b9db" (UID: "eb8b3540-eb80-42dc-b9a5-e0705554b9db"). InnerVolumeSpecName "kube-api-access-586tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:12.764266 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.764158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321f9c92-f813-452c-92de-0e5cbf57e62d-kube-api-access-6gts4" (OuterVolumeSpecName: "kube-api-access-6gts4") pod "321f9c92-f813-452c-92de-0e5cbf57e62d" (UID: "321f9c92-f813-452c-92de-0e5cbf57e62d"). InnerVolumeSpecName "kube-api-access-6gts4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:12.785749 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.785720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:12.862607 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.862574    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gts4\" (UniqueName: \"kubernetes.io/projected/321f9c92-f813-452c-92de-0e5cbf57e62d-kube-api-access-6gts4\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:12.862607 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:12.862604    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-586tf\" (UniqueName: \"kubernetes.io/projected/eb8b3540-eb80-42dc-b9a5-e0705554b9db-kube-api-access-586tf\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:13.158236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.158202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:13.259614 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.259584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:13.563263 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.563224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:13.572692 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.572656    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"
Sep 10 23:51:13.572692 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.572673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d" event={"ID":"321f9c92-f813-452c-92de-0e5cbf57e62d","Type":"ContainerDied","Data":"d4cd5e09b0d921004d5bb5ff0a3e7f5f567fc52cf3fe32d26f1189d17cf056b2"}
Sep 10 23:51:13.572893 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.572744    2578 scope.go:117] "RemoveContainer" containerID="15f1825e8a3c0303d6dad01c5a90bc588b7fa7aeab9aac8c0e55bde6cc1df05a"
Sep 10 23:51:13.574227 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.574210    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"
Sep 10 23:51:13.574340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.574209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb" event={"ID":"eb8b3540-eb80-42dc-b9a5-e0705554b9db","Type":"ContainerDied","Data":"551e78614e20a0bd160474dedf4919562aa255e05d71be3e97b7d8eb4e6dd1b9"}
Sep 10 23:51:13.584350 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.584323    2578 scope.go:117] "RemoveContainer" containerID="add032a1cfd766849213a302098a3e0400c31b67f4f07dde54a2a2dceb0da945"
Sep 10 23:51:13.601425 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.601403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"]
Sep 10 23:51:13.607375 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.607349    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-113befcd-687856cd-d68nb"]
Sep 10 23:51:13.625448 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.624532    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"]
Sep 10 23:51:13.627693 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.627672    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-113befcd-54b46d6998-4b29d"]
Sep 10 23:51:13.694175 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:13.694094    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:14.063955 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:14.063919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:14.535340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:14.535310    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="321f9c92-f813-452c-92de-0e5cbf57e62d" path="/var/lib/kubelet/pods/321f9c92-f813-452c-92de-0e5cbf57e62d/volumes"
Sep 10 23:51:14.535685 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:14.535672    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8b3540-eb80-42dc-b9a5-e0705554b9db" path="/var/lib/kubelet/pods/eb8b3540-eb80-42dc-b9a5-e0705554b9db/volumes"
Sep 10 23:51:14.985847 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:14.985757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:15.870152 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:15.870120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:16.552163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:16.552129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:16.655745 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:16.655692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:16.757011 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:16.756975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:17.576466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:17.576423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:18.430241 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:18.430207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:19.172821 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.172784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:19.271308 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.271253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:19.483496 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.483419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:19.714977 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.714942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:19.815204 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.815175    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"]
Sep 10 23:51:19.815615 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.815600    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8b3540-eb80-42dc-b9a5-e0705554b9db" containerName="payload-pre-processing"
Sep 10 23:51:19.815615 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.815616    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8b3540-eb80-42dc-b9a5-e0705554b9db" containerName="payload-pre-processing"
Sep 10 23:51:19.815709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.815647    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321f9c92-f813-452c-92de-0e5cbf57e62d" containerName="payload-processing"
Sep 10 23:51:19.815709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.815654    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="321f9c92-f813-452c-92de-0e5cbf57e62d" containerName="payload-processing"
Sep 10 23:51:19.815785 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.815731    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8b3540-eb80-42dc-b9a5-e0705554b9db" containerName="payload-pre-processing"
Sep 10 23:51:19.815785 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.815740    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="321f9c92-f813-452c-92de-0e5cbf57e62d" containerName="payload-processing"
Sep 10 23:51:19.817762 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.817726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:19.820269 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.820251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:19.823026 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.823008    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9566fcb0-dockercfg-nr6n4\""
Sep 10 23:51:19.823122 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.823058    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9566fcb0\""
Sep 10 23:51:19.829653 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.829631    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"]
Sep 10 23:51:19.857641 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.857611    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"]
Sep 10 23:51:19.860998 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.860977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:19.872547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.872523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"]
Sep 10 23:51:19.926830 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.926803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jdq\" (UniqueName: \"kubernetes.io/projected/058d19a2-f52f-44de-996d-27afcb7dd9ea-kube-api-access-f4jdq\") pod \"payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:19.926943 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:19.926835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058d19a2-f52f-44de-996d-27afcb7dd9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:20.027716 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.027681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jdq\" (UniqueName: \"kubernetes.io/projected/058d19a2-f52f-44de-996d-27afcb7dd9ea-kube-api-access-f4jdq\") pod \"payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:20.027862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.027727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc898\" (UniqueName: \"kubernetes.io/projected/3df8de43-55cd-4871-a3c7-1ac442e69172-kube-api-access-wc898\") pod \"payload-processing-e2e-ait-9566fcb0-657df9845-lq59g\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") " pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.027862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.027762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058d19a2-f52f-44de-996d-27afcb7dd9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:20.027862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.027845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df8de43-55cd-4871-a3c7-1ac442e69172-plugins-config-volume\") pod \"payload-processing-e2e-ait-9566fcb0-657df9845-lq59g\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") " pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.028514 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.028495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058d19a2-f52f-44de-996d-27afcb7dd9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:20.035730 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.035693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jdq\" (UniqueName: \"kubernetes.io/projected/058d19a2-f52f-44de-996d-27afcb7dd9ea-kube-api-access-f4jdq\") pod \"payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:20.128970 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.128910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df8de43-55cd-4871-a3c7-1ac442e69172-plugins-config-volume\") pod \"payload-processing-e2e-ait-9566fcb0-657df9845-lq59g\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") " pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.128970 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.128956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc898\" (UniqueName: \"kubernetes.io/projected/3df8de43-55cd-4871-a3c7-1ac442e69172-kube-api-access-wc898\") pod \"payload-processing-e2e-ait-9566fcb0-657df9845-lq59g\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") " pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.129675 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.129655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df8de43-55cd-4871-a3c7-1ac442e69172-plugins-config-volume\") pod \"payload-processing-e2e-ait-9566fcb0-657df9845-lq59g\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") " pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.131842 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.131819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:20.137426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.137404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc898\" (UniqueName: \"kubernetes.io/projected/3df8de43-55cd-4871-a3c7-1ac442e69172-kube-api-access-wc898\") pod \"payload-processing-e2e-ait-9566fcb0-657df9845-lq59g\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") " pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.172248 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.172213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.402611 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.402586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"]
Sep 10 23:51:20.404652 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:51:20.404623    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058d19a2_f52f_44de_996d_27afcb7dd9ea.slice/crio-0921c1d9a55e3a1d60d73677bf4af7936d8e82d1bcec9b10a837a9149314d9a2 WatchSource:0}: Error finding container 0921c1d9a55e3a1d60d73677bf4af7936d8e82d1bcec9b10a837a9149314d9a2: Status 404 returned error can't find the container with id 0921c1d9a55e3a1d60d73677bf4af7936d8e82d1bcec9b10a837a9149314d9a2
Sep 10 23:51:20.454054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.454026    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"]
Sep 10 23:51:20.456339 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:51:20.456315    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df8de43_55cd_4871_a3c7_1ac442e69172.slice/crio-315a4f8b9c7ef08321a43dc7e046b34d267f7a94d91d477b83e79fa2cec98b58 WatchSource:0}: Error finding container 315a4f8b9c7ef08321a43dc7e046b34d267f7a94d91d477b83e79fa2cec98b58: Status 404 returned error can't find the container with id 315a4f8b9c7ef08321a43dc7e046b34d267f7a94d91d477b83e79fa2cec98b58
Sep 10 23:51:20.614451 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.614411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5" event={"ID":"058d19a2-f52f-44de-996d-27afcb7dd9ea","Type":"ContainerStarted","Data":"002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d"}
Sep 10 23:51:20.614619 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.614486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5" event={"ID":"058d19a2-f52f-44de-996d-27afcb7dd9ea","Type":"ContainerStarted","Data":"0921c1d9a55e3a1d60d73677bf4af7936d8e82d1bcec9b10a837a9149314d9a2"}
Sep 10 23:51:20.614619 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.614515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:20.616095 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.616070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g" event={"ID":"3df8de43-55cd-4871-a3c7-1ac442e69172","Type":"ContainerStarted","Data":"24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b"}
Sep 10 23:51:20.616206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.616113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g" event={"ID":"3df8de43-55cd-4871-a3c7-1ac442e69172","Type":"ContainerStarted","Data":"315a4f8b9c7ef08321a43dc7e046b34d267f7a94d91d477b83e79fa2cec98b58"}
Sep 10 23:51:20.616206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.616152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:20.632943 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.632886    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5" podStartSLOduration=1.632867233 podStartE2EDuration="1.632867233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:20.631353035 +0000 UTC m=+922.587584842" watchObservedRunningTime="2026-09-10 23:51:20.632867233 +0000 UTC m=+922.589099036"
Sep 10 23:51:20.650408 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:20.650369    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g" podStartSLOduration=1.650355926 podStartE2EDuration="1.650355926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:20.647844533 +0000 UTC m=+922.604076336" watchObservedRunningTime="2026-09-10 23:51:20.650355926 +0000 UTC m=+922.606587729"
Sep 10 23:51:26.657094 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:26.657048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:26.757990 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:26.757952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:27.405540 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:27.405504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:51:31.624448 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:31.624411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:31.624870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:31.624588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:40.996677 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:40.996642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:51:41.372462 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:41.372424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:42.695498 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:42.695458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:42.857568 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:42.857504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:43.885177 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:43.885141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:44.858103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:44.858065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:45.622758 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:45.622717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:46.366733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:46.366693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:48.183404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:48.183369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:48.892217 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:48.892179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:48.920779 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:48.920734    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"]
Sep 10 23:51:48.920987 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:48.920961    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g" podUID="3df8de43-55cd-4871-a3c7-1ac442e69172" containerName="payload-processing" containerID="cri-o://24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b" gracePeriod=30
Sep 10 23:51:48.925842 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:48.925807    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"]
Sep 10 23:51:48.926244 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:48.926210    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5" podUID="058d19a2-f52f-44de-996d-27afcb7dd9ea" containerName="payload-pre-processing" containerID="cri-o://002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d" gracePeriod=30
Sep 10 23:51:48.969214 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:48.969185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:49.039774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.039740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:49.109711 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.109675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:49.210312 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.210267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:49.213069 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.213048    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:49.273849 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.273822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4jdq\" (UniqueName: \"kubernetes.io/projected/058d19a2-f52f-44de-996d-27afcb7dd9ea-kube-api-access-f4jdq\") pod \"058d19a2-f52f-44de-996d-27afcb7dd9ea\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") "
Sep 10 23:51:49.274036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.273883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058d19a2-f52f-44de-996d-27afcb7dd9ea-plugins-config-volume\") pod \"058d19a2-f52f-44de-996d-27afcb7dd9ea\" (UID: \"058d19a2-f52f-44de-996d-27afcb7dd9ea\") "
Sep 10 23:51:49.274036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.273920    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc898\" (UniqueName: \"kubernetes.io/projected/3df8de43-55cd-4871-a3c7-1ac442e69172-kube-api-access-wc898\") pod \"3df8de43-55cd-4871-a3c7-1ac442e69172\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") "
Sep 10 23:51:49.274036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.274016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df8de43-55cd-4871-a3c7-1ac442e69172-plugins-config-volume\") pod \"3df8de43-55cd-4871-a3c7-1ac442e69172\" (UID: \"3df8de43-55cd-4871-a3c7-1ac442e69172\") "
Sep 10 23:51:49.274293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.274245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058d19a2-f52f-44de-996d-27afcb7dd9ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "058d19a2-f52f-44de-996d-27afcb7dd9ea" (UID: "058d19a2-f52f-44de-996d-27afcb7dd9ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:49.274395 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.274337    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058d19a2-f52f-44de-996d-27afcb7dd9ea-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:49.274457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.274430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df8de43-55cd-4871-a3c7-1ac442e69172-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3df8de43-55cd-4871-a3c7-1ac442e69172" (UID: "3df8de43-55cd-4871-a3c7-1ac442e69172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:49.276135 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.276107    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df8de43-55cd-4871-a3c7-1ac442e69172-kube-api-access-wc898" (OuterVolumeSpecName: "kube-api-access-wc898") pod "3df8de43-55cd-4871-a3c7-1ac442e69172" (UID: "3df8de43-55cd-4871-a3c7-1ac442e69172"). InnerVolumeSpecName "kube-api-access-wc898". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:49.276247 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.276144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058d19a2-f52f-44de-996d-27afcb7dd9ea-kube-api-access-f4jdq" (OuterVolumeSpecName: "kube-api-access-f4jdq") pod "058d19a2-f52f-44de-996d-27afcb7dd9ea" (UID: "058d19a2-f52f-44de-996d-27afcb7dd9ea"). InnerVolumeSpecName "kube-api-access-f4jdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:49.375549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.375524    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc898\" (UniqueName: \"kubernetes.io/projected/3df8de43-55cd-4871-a3c7-1ac442e69172-kube-api-access-wc898\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:49.375549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.375548    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df8de43-55cd-4871-a3c7-1ac442e69172-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:49.375716 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.375559    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4jdq\" (UniqueName: \"kubernetes.io/projected/058d19a2-f52f-44de-996d-27afcb7dd9ea-kube-api-access-f4jdq\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:51:49.572115 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.572085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:49.762119 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.762082    2578 generic.go:356] "Generic (PLEG): container finished" podID="3df8de43-55cd-4871-a3c7-1ac442e69172" containerID="24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b" exitCode=0
Sep 10 23:51:49.762267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.762128    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"
Sep 10 23:51:49.762267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.762171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g" event={"ID":"3df8de43-55cd-4871-a3c7-1ac442e69172","Type":"ContainerDied","Data":"24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b"}
Sep 10 23:51:49.762267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.762226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g" event={"ID":"3df8de43-55cd-4871-a3c7-1ac442e69172","Type":"ContainerDied","Data":"315a4f8b9c7ef08321a43dc7e046b34d267f7a94d91d477b83e79fa2cec98b58"}
Sep 10 23:51:49.762267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.762246    2578 scope.go:117] "RemoveContainer" containerID="24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b"
Sep 10 23:51:49.763628 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.763606    2578 generic.go:356] "Generic (PLEG): container finished" podID="058d19a2-f52f-44de-996d-27afcb7dd9ea" containerID="002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d" exitCode=0
Sep 10 23:51:49.763725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.763649    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"
Sep 10 23:51:49.763725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.763684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5" event={"ID":"058d19a2-f52f-44de-996d-27afcb7dd9ea","Type":"ContainerDied","Data":"002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d"}
Sep 10 23:51:49.763839 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.763739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5" event={"ID":"058d19a2-f52f-44de-996d-27afcb7dd9ea","Type":"ContainerDied","Data":"0921c1d9a55e3a1d60d73677bf4af7936d8e82d1bcec9b10a837a9149314d9a2"}
Sep 10 23:51:49.771360 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.771338    2578 scope.go:117] "RemoveContainer" containerID="24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b"
Sep 10 23:51:49.771646 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:51:49.771619    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b\": container with ID starting with 24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b not found: ID does not exist" containerID="24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b"
Sep 10 23:51:49.771867 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.771656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b"} err="failed to get container status \"24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b\": rpc error: code = NotFound desc = could not find container \"24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b\": container with ID starting with 24eeb9b62d18d8456670fdbcdc969d94c082e10ac785a9a8f316f77e2d1b803b not found: ID does not exist"
Sep 10 23:51:49.771867 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.771678    2578 scope.go:117] "RemoveContainer" containerID="002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d"
Sep 10 23:51:49.779309 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.779294    2578 scope.go:117] "RemoveContainer" containerID="002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d"
Sep 10 23:51:49.779555 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:51:49.779538    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d\": container with ID starting with 002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d not found: ID does not exist" containerID="002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d"
Sep 10 23:51:49.779601 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.779561    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d"} err="failed to get container status \"002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d\": rpc error: code = NotFound desc = could not find container \"002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d\": container with ID starting with 002f01a16b842df57d8d5e4f9e46b10f93ae101965c9abd92ae35c1f7f4dc50d not found: ID does not exist"
Sep 10 23:51:49.791180 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.791160    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"]
Sep 10 23:51:49.796149 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.796128    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9566fcb0-657df9845-lq59g"]
Sep 10 23:51:49.810206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.810183    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"]
Sep 10 23:51:49.814250 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:49.814231    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9566fcb0-7c4fcbd8c8-8jfz5"]
Sep 10 23:51:50.212588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:50.212551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:50.322979 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:50.322949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:50.535386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:50.535349    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058d19a2-f52f-44de-996d-27afcb7dd9ea" path="/var/lib/kubelet/pods/058d19a2-f52f-44de-996d-27afcb7dd9ea/volumes"
Sep 10 23:51:50.535905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:50.535886    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df8de43-55cd-4871-a3c7-1ac442e69172" path="/var/lib/kubelet/pods/3df8de43-55cd-4871-a3c7-1ac442e69172/volumes"
Sep 10 23:51:50.633266 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:50.633233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:50.937703 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:50.937609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:51.372573 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:51.372538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:51.466322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:51.466288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:52.101602 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:52.101553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:53.123717 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:53.123678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:53.723140 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:53.723109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:53.821837 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:53.821799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:54.183754 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:54.183716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:54.994606 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:54.994573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:55.789234 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:55.789198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:51:55.869612 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:55.869577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:56.582890 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:56.582858    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:56.682585 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:56.682557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:57.062352 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.062314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:57.162141 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.162106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:57.193350 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.193318    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"]
Sep 10 23:51:57.193809 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.193791    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df8de43-55cd-4871-a3c7-1ac442e69172" containerName="payload-processing"
Sep 10 23:51:57.193870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.193811    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df8de43-55cd-4871-a3c7-1ac442e69172" containerName="payload-processing"
Sep 10 23:51:57.193870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.193825    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058d19a2-f52f-44de-996d-27afcb7dd9ea" containerName="payload-pre-processing"
Sep 10 23:51:57.193870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.193833    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="058d19a2-f52f-44de-996d-27afcb7dd9ea" containerName="payload-pre-processing"
Sep 10 23:51:57.193966 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.193904    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df8de43-55cd-4871-a3c7-1ac442e69172" containerName="payload-processing"
Sep 10 23:51:57.193966 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.193913    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="058d19a2-f52f-44de-996d-27afcb7dd9ea" containerName="payload-pre-processing"
Sep 10 23:51:57.197050 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.197034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.199633 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.199612    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a105b100-dockercfg-8wvkh\""
Sep 10 23:51:57.199744 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.199656    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a105b100\""
Sep 10 23:51:57.206431 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.206408    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"]
Sep 10 23:51:57.243143 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.243111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n98h\" (UniqueName: \"kubernetes.io/projected/81183c3c-74d0-4bca-9272-0605d85b6c21-kube-api-access-4n98h\") pod \"payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") " pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.243319 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.243250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81183c3c-74d0-4bca-9272-0605d85b6c21-plugins-config-volume\") pod \"payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") " pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.344243 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.344158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n98h\" (UniqueName: \"kubernetes.io/projected/81183c3c-74d0-4bca-9272-0605d85b6c21-kube-api-access-4n98h\") pod \"payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") " pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.344415 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.344243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81183c3c-74d0-4bca-9272-0605d85b6c21-plugins-config-volume\") pod \"payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") " pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.344930 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.344908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81183c3c-74d0-4bca-9272-0605d85b6c21-plugins-config-volume\") pod \"payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") " pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.353241 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.353209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n98h\" (UniqueName: \"kubernetes.io/projected/81183c3c-74d0-4bca-9272-0605d85b6c21-kube-api-access-4n98h\") pod \"payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") " pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.508875 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.508831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:57.778356 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.778330    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"]
Sep 10 23:51:57.780459 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:51:57.780430    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81183c3c_74d0_4bca_9272_0605d85b6c21.slice/crio-9333c8c63d8a67ec2b7de5d9fe305d095ebe500fee289b1c7c926f01d2c0ada4 WatchSource:0}: Error finding container 9333c8c63d8a67ec2b7de5d9fe305d095ebe500fee289b1c7c926f01d2c0ada4: Status 404 returned error can't find the container with id 9333c8c63d8a67ec2b7de5d9fe305d095ebe500fee289b1c7c926f01d2c0ada4
Sep 10 23:51:57.804161 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:57.804137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm" event={"ID":"81183c3c-74d0-4bca-9272-0605d85b6c21","Type":"ContainerStarted","Data":"9333c8c63d8a67ec2b7de5d9fe305d095ebe500fee289b1c7c926f01d2c0ada4"}
Sep 10 23:51:58.810099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:58.810065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm" event={"ID":"81183c3c-74d0-4bca-9272-0605d85b6c21","Type":"ContainerStarted","Data":"f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e"}
Sep 10 23:51:58.810508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:58.810132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:51:58.827178 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:58.827128    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm" podStartSLOduration=1.827112327 podStartE2EDuration="1.827112327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:58.826334129 +0000 UTC m=+960.782565930" watchObservedRunningTime="2026-09-10 23:51:58.827112327 +0000 UTC m=+960.783344129"
Sep 10 23:51:59.062666 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:51:59.062627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:09.436383 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.436350    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"]
Sep 10 23:52:09.441115 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.441094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.445133 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.445102    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b3de77\"/\"default-dockercfg-8d42l\""
Sep 10 23:52:09.445345 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.445127    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b3de77\"/\"openshift-service-ca.crt\""
Sep 10 23:52:09.445345 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.445188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b3de77\"/\"kube-root-ca.crt\""
Sep 10 23:52:09.445345 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.445126    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b3de77\"/\"autores-model-b3de77-kserve-self-signed-certs\""
Sep 10 23:52:09.448189 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.448165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"]
Sep 10 23:52:09.554498 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.554470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tmp-dir\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.554657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.554514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks92j\" (UniqueName: \"kubernetes.io/projected/29c75ca5-f97f-48ea-8aed-43c0119ef88e-kube-api-access-ks92j\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.554657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.554534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-model-cache\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.554657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.554608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-home\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.554813 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.554664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tls-certs\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.554813 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.554694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-dshm\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.556642 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.556617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:09.655479 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.655454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tls-certs\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.655616 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.655487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-dshm\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.655616 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.655569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tmp-dir\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.655726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.655616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks92j\" (UniqueName: \"kubernetes.io/projected/29c75ca5-f97f-48ea-8aed-43c0119ef88e-kube-api-access-ks92j\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.655726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.655646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-model-cache\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.655832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.655739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-home\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.656057 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.656030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tmp-dir\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.656178 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.656072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-home\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.656178 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.656130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-model-cache\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.657730 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.657696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-dshm\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.657878 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.657841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tls-certs\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.664046 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.664023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks92j\" (UniqueName: \"kubernetes.io/projected/29c75ca5-f97f-48ea-8aed-43c0119ef88e-kube-api-access-ks92j\") pod \"autores-model-b3de77-kserve-6f94d9c95-8zwxw\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") " pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.751737 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.751671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:09.817162 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:09.817137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:52:10.038567 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:10.038539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"]
Sep 10 23:52:10.039598 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:52:10.039573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29c75ca5_f97f_48ea_8aed_43c0119ef88e.slice/crio-d3457282ee7d581940c39e3077c421a35d52478132e3f3386dc030191a403f14 WatchSource:0}: Error finding container d3457282ee7d581940c39e3077c421a35d52478132e3f3386dc030191a403f14: Status 404 returned error can't find the container with id d3457282ee7d581940c39e3077c421a35d52478132e3f3386dc030191a403f14
Sep 10 23:52:10.292214 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:10.292136    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:52:10.876487 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:10.876443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw" event={"ID":"29c75ca5-f97f-48ea-8aed-43c0119ef88e","Type":"ContainerStarted","Data":"4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34"}
Sep 10 23:52:10.876873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:10.876500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw" event={"ID":"29c75ca5-f97f-48ea-8aed-43c0119ef88e","Type":"ContainerStarted","Data":"d3457282ee7d581940c39e3077c421a35d52478132e3f3386dc030191a403f14"}
Sep 10 23:52:10.897331 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:10.897259    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw" podStartSLOduration=1.897243612 podStartE2EDuration="1.897243612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:10.896438164 +0000 UTC m=+972.852669969" watchObservedRunningTime="2026-09-10 23:52:10.897243612 +0000 UTC m=+972.853475415"
Sep 10 23:52:17.475265 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:17.475220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:19.262654 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:19.262620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:19.752377 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:19.752269    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:19.752377 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:19.752352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:19.765492 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:19.765461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:19.931843 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:19.931805    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:52:20.139088 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:20.139048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:20.980669 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:20.980583    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:21.890558 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:21.890524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:22.653190 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:22.653158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:23.895320 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:23.895264    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:52:25.836961 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:25.836919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:25.850947 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:25.850920    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"]
Sep 10 23:52:25.851153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:25.851121    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm" podUID="81183c3c-74d0-4bca-9272-0605d85b6c21" containerName="payload-processing" containerID="cri-o://f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e" gracePeriod=30
Sep 10 23:52:25.937300 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:25.937232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:26.016386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.016355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:26.078231 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.078196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:26.106728 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.106676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:52:26.167396 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.167360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:26.213661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.213635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n98h\" (UniqueName: \"kubernetes.io/projected/81183c3c-74d0-4bca-9272-0605d85b6c21-kube-api-access-4n98h\") pod \"81183c3c-74d0-4bca-9272-0605d85b6c21\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") "
Sep 10 23:52:26.213806 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.213691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81183c3c-74d0-4bca-9272-0605d85b6c21-plugins-config-volume\") pod \"81183c3c-74d0-4bca-9272-0605d85b6c21\" (UID: \"81183c3c-74d0-4bca-9272-0605d85b6c21\") "
Sep 10 23:52:26.214194 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.214126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81183c3c-74d0-4bca-9272-0605d85b6c21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81183c3c-74d0-4bca-9272-0605d85b6c21" (UID: "81183c3c-74d0-4bca-9272-0605d85b6c21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:52:26.215762 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.215739    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81183c3c-74d0-4bca-9272-0605d85b6c21-kube-api-access-4n98h" (OuterVolumeSpecName: "kube-api-access-4n98h") pod "81183c3c-74d0-4bca-9272-0605d85b6c21" (UID: "81183c3c-74d0-4bca-9272-0605d85b6c21"). InnerVolumeSpecName "kube-api-access-4n98h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:52:26.315143 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.315113    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n98h\" (UniqueName: \"kubernetes.io/projected/81183c3c-74d0-4bca-9272-0605d85b6c21-kube-api-access-4n98h\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:52:26.315143 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.315139    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81183c3c-74d0-4bca-9272-0605d85b6c21-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:52:26.459112 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.459032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:26.838078 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.838046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:26.930288 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.930246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:26.957365 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.957331    2578 generic.go:356] "Generic (PLEG): container finished" podID="81183c3c-74d0-4bca-9272-0605d85b6c21" containerID="f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e" exitCode=0
Sep 10 23:52:26.957513 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.957371    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"
Sep 10 23:52:26.957513 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.957414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm" event={"ID":"81183c3c-74d0-4bca-9272-0605d85b6c21","Type":"ContainerDied","Data":"f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e"}
Sep 10 23:52:26.957513 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.957464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm" event={"ID":"81183c3c-74d0-4bca-9272-0605d85b6c21","Type":"ContainerDied","Data":"9333c8c63d8a67ec2b7de5d9fe305d095ebe500fee289b1c7c926f01d2c0ada4"}
Sep 10 23:52:26.957513 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.957488    2578 scope.go:117] "RemoveContainer" containerID="f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e"
Sep 10 23:52:26.965836 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.965818    2578 scope.go:117] "RemoveContainer" containerID="f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e"
Sep 10 23:52:26.966090 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:52:26.966070    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e\": container with ID starting with f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e not found: ID does not exist" containerID="f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e"
Sep 10 23:52:26.966159 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.966104    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e"} err="failed to get container status \"f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e\": rpc error: code = NotFound desc = could not find container \"f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e\": container with ID starting with f1374f0356f5687c10228c0a09c36ae2e7d83cba761a52708e7fd6177afa1e7e not found: ID does not exist"
Sep 10 23:52:26.974047 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.974026    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"]
Sep 10 23:52:26.978717 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:26.978696    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a105b100-75f6ffcbbf-dpwcm"]
Sep 10 23:52:27.035588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:27.035553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:27.239356 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:27.239255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:27.943570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:27.943519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:28.331023 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:28.330984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:28.540003 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:28.539968    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81183c3c-74d0-4bca-9272-0605d85b6c21" path="/var/lib/kubelet/pods/81183c3c-74d0-4bca-9272-0605d85b6c21/volumes"
Sep 10 23:52:28.781510 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:28.781479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:30.157059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.157013    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"]
Sep 10 23:52:30.161293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.159495    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81183c3c-74d0-4bca-9272-0605d85b6c21" containerName="payload-processing"
Sep 10 23:52:30.161293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.159524    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81183c3c-74d0-4bca-9272-0605d85b6c21" containerName="payload-processing"
Sep 10 23:52:30.161293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.159643    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81183c3c-74d0-4bca-9272-0605d85b6c21" containerName="payload-processing"
Sep 10 23:52:30.164833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.164804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.167810 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.167783    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4c5699\"/\"openshift-service-ca.crt\""
Sep 10 23:52:30.168993 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.168970    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4c5699\"/\"kube-root-ca.crt\""
Sep 10 23:52:30.169105 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.169014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4c5699\"/\"default-dockercfg-7h854\""
Sep 10 23:52:30.169420 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.169400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4c5699\"/\"test-model-4c5699-kserve-self-signed-certs\""
Sep 10 23:52:30.175644 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.175623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"]
Sep 10 23:52:30.178120 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.178096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:30.262595 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.262569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-tmp-dir\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.262715 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.262615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e664-e007-462f-987b-c9820e1da1a1-tls-certs\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.262715 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.262634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-dshm\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.262799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.262729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-model-cache\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.262799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.262762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f94r\" (UniqueName: \"kubernetes.io/projected/a3e5e664-e007-462f-987b-c9820e1da1a1-kube-api-access-5f94r\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.262799 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.262780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-home\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.363960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.363928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e664-e007-462f-987b-c9820e1da1a1-tls-certs\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.363970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-dshm\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.364037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-model-cache\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.364074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f94r\" (UniqueName: \"kubernetes.io/projected/a3e5e664-e007-462f-987b-c9820e1da1a1-kube-api-access-5f94r\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.364105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-home\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364400 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.364169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-tmp-dir\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364482 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.364457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-model-cache\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364621 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.364602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-home\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.364687 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.364633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-tmp-dir\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.366883 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.366849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-dshm\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.367005 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.366990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e664-e007-462f-987b-c9820e1da1a1-tls-certs\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.376063 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.376031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f94r\" (UniqueName: \"kubernetes.io/projected/a3e5e664-e007-462f-987b-c9820e1da1a1-kube-api-access-5f94r\") pod \"test-model-4c5699-kserve-5bd6c99d4f-lppld\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") " pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.475412 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.475321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:30.744896 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.744867    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"]
Sep 10 23:52:30.746327 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:52:30.746302    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e5e664_e007_462f_987b_c9820e1da1a1.slice/crio-327fbd11fc575f0745ae2138d675d251004720f2f6f0be3ad95fc40dbb69acf9 WatchSource:0}: Error finding container 327fbd11fc575f0745ae2138d675d251004720f2f6f0be3ad95fc40dbb69acf9: Status 404 returned error can't find the container with id 327fbd11fc575f0745ae2138d675d251004720f2f6f0be3ad95fc40dbb69acf9
Sep 10 23:52:30.979163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.979127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld" event={"ID":"a3e5e664-e007-462f-987b-c9820e1da1a1","Type":"ContainerStarted","Data":"fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd"}
Sep 10 23:52:30.979347 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.979176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld" event={"ID":"a3e5e664-e007-462f-987b-c9820e1da1a1","Type":"ContainerStarted","Data":"327fbd11fc575f0745ae2138d675d251004720f2f6f0be3ad95fc40dbb69acf9"}
Sep 10 23:52:30.998085 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:30.998002    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld" podStartSLOduration=0.997985587 podStartE2EDuration="997.985587ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:30.997925483 +0000 UTC m=+992.954157289" watchObservedRunningTime="2026-09-10 23:52:30.997985587 +0000 UTC m=+992.954217389"
Sep 10 23:52:34.386855 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:34.386818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:34.453604 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:34.453554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:34.552614 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:34.552582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:34.648165 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:34.648068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:34.745743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:34.745707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:35.150343 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:35.150299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:39.699206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:39.699170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:52:40.475719 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:40.475683    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:40.475893 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:40.475739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:40.488523 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:40.488499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:41.052498 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:41.052467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:52:42.461569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:42.461534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:43.291214 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:43.291172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:44.056369 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:44.056329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:44.919658 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:44.919622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:52.056994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.056959    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"]
Sep 10 23:52:52.084345 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.084308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:52.094109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.094073    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld" podUID="46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" containerName="maas-api" containerID="cri-o://66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab" gracePeriod=30
Sep 10 23:52:52.179190 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.179156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:52.258854 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.258813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:52.355964 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.355931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:52.359232 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.359211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:52:52.463696 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.463663    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl6dx\" (UniqueName: \"kubernetes.io/projected/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-kube-api-access-nl6dx\") pod \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") "
Sep 10 23:52:52.463864 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.463704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls\") pod \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") "
Sep 10 23:52:52.463864 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.463854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-openshift-service-ca\") pod \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") "
Sep 10 23:52:52.463941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.463927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-metrics-certs\") pod \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\" (UID: \"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a\") "
Sep 10 23:52:52.464237 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.464200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" (UID: "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:52:52.465879 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.465855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" (UID: "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:52:52.465971 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.465929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" (UID: "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:52:52.465971 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.465958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-kube-api-access-nl6dx" (OuterVolumeSpecName: "kube-api-access-nl6dx") pod "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" (UID: "46d1829b-1ac4-4c46-8da0-44c2a1bdd71a"). InnerVolumeSpecName "kube-api-access-nl6dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:52:52.564699 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.564669    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-openshift-service-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.564699 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.564696    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-metrics-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.564868 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.564706    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl6dx\" (UniqueName: \"kubernetes.io/projected/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-kube-api-access-nl6dx\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.564868 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.564714    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a-maas-api-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.723254 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.723166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:52.823909 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:52.823872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:53.100330 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.100299    2578 generic.go:356] "Generic (PLEG): container finished" podID="46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" containerID="66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab" exitCode=0
Sep 10 23:52:53.100724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.100362    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"
Sep 10 23:52:53.100724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.100364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld" event={"ID":"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a","Type":"ContainerDied","Data":"66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab"}
Sep 10 23:52:53.100724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.100484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld" event={"ID":"46d1829b-1ac4-4c46-8da0-44c2a1bdd71a","Type":"ContainerDied","Data":"e1e9ed556d823eee6984dba3b04c56b6fa9852a525402e8e8744efb3ce8f12e3"}
Sep 10 23:52:53.100724 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.100512    2578 scope.go:117] "RemoveContainer" containerID="66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab"
Sep 10 23:52:53.109101 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.109085    2578 scope.go:117] "RemoveContainer" containerID="66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab"
Sep 10 23:52:53.109496 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:52:53.109477    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab\": container with ID starting with 66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab not found: ID does not exist" containerID="66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab"
Sep 10 23:52:53.109593 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.109502    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab"} err="failed to get container status \"66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab\": rpc error: code = NotFound desc = could not find container \"66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab\": container with ID starting with 66d5be65dcbc376124d91517d72f91fe52ea60cb5312b513c7a02c97278aa1ab not found: ID does not exist"
Sep 10 23:52:53.125300 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.125255    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"]
Sep 10 23:52:53.129468 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.129444    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b3de77-5fdf8f46c9-9vzld"]
Sep 10 23:52:53.598508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:53.598476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:54.457842 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:54.457805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:54.535296 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:54.535240    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" path="/var/lib/kubelet/pods/46d1829b-1ac4-4c46-8da0-44c2a1bdd71a/volumes"
Sep 10 23:52:55.416744 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:55.416706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:56.786742 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:56.786704    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:52:57.708644 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:57.708605    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"]
Sep 10 23:52:57.803029 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:57.802985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:59.491304 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:52:59.491250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:01.084793 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:01.084756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:04.248188 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:04.248149    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"]
Sep 10 23:53:04.270708 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:04.270673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:07.161015 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:07.160969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:07.268457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:07.268428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:07.377900 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:07.377867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:07.478031 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:07.477951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:10.995474 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:10.995435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:53:12.731018 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:12.730970    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw" podUID="29c75ca5-f97f-48ea-8aed-43c0119ef88e" containerName="main" containerID="cri-o://4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34" gracePeriod=45
Sep 10 23:53:12.978887 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:12.978864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:53:13.155880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.155846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-dshm\") pod \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") "
Sep 10 23:53:13.156056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.155895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-home\") pod \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") "
Sep 10 23:53:13.156056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.155950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks92j\" (UniqueName: \"kubernetes.io/projected/29c75ca5-f97f-48ea-8aed-43c0119ef88e-kube-api-access-ks92j\") pod \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") "
Sep 10 23:53:13.156056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.156017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tls-certs\") pod \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") "
Sep 10 23:53:13.156056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.156037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-model-cache\") pod \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") "
Sep 10 23:53:13.156056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.156055    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tmp-dir\") pod \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\" (UID: \"29c75ca5-f97f-48ea-8aed-43c0119ef88e\") "
Sep 10 23:53:13.156335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.156177    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-home" (OuterVolumeSpecName: "home") pod "29c75ca5-f97f-48ea-8aed-43c0119ef88e" (UID: "29c75ca5-f97f-48ea-8aed-43c0119ef88e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:13.157037 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.156638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-model-cache" (OuterVolumeSpecName: "model-cache") pod "29c75ca5-f97f-48ea-8aed-43c0119ef88e" (UID: "29c75ca5-f97f-48ea-8aed-43c0119ef88e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:13.157037 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.156893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29c75ca5-f97f-48ea-8aed-43c0119ef88e" (UID: "29c75ca5-f97f-48ea-8aed-43c0119ef88e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:13.157228 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.157157    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:13.160318 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.159232    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29c75ca5-f97f-48ea-8aed-43c0119ef88e" (UID: "29c75ca5-f97f-48ea-8aed-43c0119ef88e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:53:13.160318 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.159238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c75ca5-f97f-48ea-8aed-43c0119ef88e-kube-api-access-ks92j" (OuterVolumeSpecName: "kube-api-access-ks92j") pod "29c75ca5-f97f-48ea-8aed-43c0119ef88e" (UID: "29c75ca5-f97f-48ea-8aed-43c0119ef88e"). InnerVolumeSpecName "kube-api-access-ks92j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:53:13.164516 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.164489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-dshm" (OuterVolumeSpecName: "dshm") pod "29c75ca5-f97f-48ea-8aed-43c0119ef88e" (UID: "29c75ca5-f97f-48ea-8aed-43c0119ef88e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:13.199583 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.199551    2578 generic.go:356] "Generic (PLEG): container finished" podID="29c75ca5-f97f-48ea-8aed-43c0119ef88e" containerID="4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34" exitCode=0
Sep 10 23:53:13.199674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.199603    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"
Sep 10 23:53:13.199674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.199639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw" event={"ID":"29c75ca5-f97f-48ea-8aed-43c0119ef88e","Type":"ContainerDied","Data":"4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34"}
Sep 10 23:53:13.199757 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.199675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw" event={"ID":"29c75ca5-f97f-48ea-8aed-43c0119ef88e","Type":"ContainerDied","Data":"d3457282ee7d581940c39e3077c421a35d52478132e3f3386dc030191a403f14"}
Sep 10 23:53:13.199757 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.199692    2578 scope.go:117] "RemoveContainer" containerID="4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34"
Sep 10 23:53:13.208748 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.208732    2578 scope.go:117] "RemoveContainer" containerID="4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34"
Sep 10 23:53:13.208995 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:53:13.208976    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34\": container with ID starting with 4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34 not found: ID does not exist" containerID="4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34"
Sep 10 23:53:13.209062 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.209003    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34"} err="failed to get container status \"4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34\": rpc error: code = NotFound desc = could not find container \"4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34\": container with ID starting with 4537bb14c56d156634a428f985ce6a981bc4d34d238a6dca2edbb577cba5ef34 not found: ID does not exist"
Sep 10 23:53:13.220842 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.220820    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"]
Sep 10 23:53:13.227084 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.227064    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b3de77/autores-model-b3de77-kserve-6f94d9c95-8zwxw"]
Sep 10 23:53:13.257616 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.257594    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:13.257616 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.257613    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:13.257736 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.257621    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:13.257736 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.257629    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c75ca5-f97f-48ea-8aed-43c0119ef88e-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:13.257736 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:13.257637    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks92j\" (UniqueName: \"kubernetes.io/projected/29c75ca5-f97f-48ea-8aed-43c0119ef88e-kube-api-access-ks92j\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:14.535056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:14.535017    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c75ca5-f97f-48ea-8aed-43c0119ef88e" path="/var/lib/kubelet/pods/29c75ca5-f97f-48ea-8aed-43c0119ef88e/volumes"
Sep 10 23:53:19.267108 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.267044    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr" podUID="a24d016f-15ec-4aed-a8c3-dfe85091cc94" containerName="main" containerID="cri-o://00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445" gracePeriod=45
Sep 10 23:53:19.519886 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.519828    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:53:19.606570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tls-certs\") pod \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") "
Sep 10 23:53:19.606717 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606589    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-dshm\") pod \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") "
Sep 10 23:53:19.606717 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-home\") pod \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") "
Sep 10 23:53:19.606717 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-model-cache\") pod \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") "
Sep 10 23:53:19.606717 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tmp-dir\") pod \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") "
Sep 10 23:53:19.606919 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bdnn\" (UniqueName: \"kubernetes.io/projected/a24d016f-15ec-4aed-a8c3-dfe85091cc94-kube-api-access-4bdnn\") pod \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\" (UID: \"a24d016f-15ec-4aed-a8c3-dfe85091cc94\") "
Sep 10 23:53:19.606997 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a24d016f-15ec-4aed-a8c3-dfe85091cc94" (UID: "a24d016f-15ec-4aed-a8c3-dfe85091cc94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:19.606997 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606963    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-model-cache" (OuterVolumeSpecName: "model-cache") pod "a24d016f-15ec-4aed-a8c3-dfe85091cc94" (UID: "a24d016f-15ec-4aed-a8c3-dfe85091cc94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:19.606997 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.606970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-home" (OuterVolumeSpecName: "home") pod "a24d016f-15ec-4aed-a8c3-dfe85091cc94" (UID: "a24d016f-15ec-4aed-a8c3-dfe85091cc94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:19.608922 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.608890    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-dshm" (OuterVolumeSpecName: "dshm") pod "a24d016f-15ec-4aed-a8c3-dfe85091cc94" (UID: "a24d016f-15ec-4aed-a8c3-dfe85091cc94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:53:19.608922 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.608900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24d016f-15ec-4aed-a8c3-dfe85091cc94-kube-api-access-4bdnn" (OuterVolumeSpecName: "kube-api-access-4bdnn") pod "a24d016f-15ec-4aed-a8c3-dfe85091cc94" (UID: "a24d016f-15ec-4aed-a8c3-dfe85091cc94"). InnerVolumeSpecName "kube-api-access-4bdnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:53:19.609103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.608943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a24d016f-15ec-4aed-a8c3-dfe85091cc94" (UID: "a24d016f-15ec-4aed-a8c3-dfe85091cc94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:53:19.708171 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.708137    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:19.708171 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.708163    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:19.708171 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.708171    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:19.708171 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.708179    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:19.708443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.708204    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24d016f-15ec-4aed-a8c3-dfe85091cc94-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:19.708443 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:19.708212    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bdnn\" (UniqueName: \"kubernetes.io/projected/a24d016f-15ec-4aed-a8c3-dfe85091cc94-kube-api-access-4bdnn\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:53:20.235263 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.235224    2578 generic.go:356] "Generic (PLEG): container finished" podID="a24d016f-15ec-4aed-a8c3-dfe85091cc94" containerID="00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445" exitCode=0
Sep 10 23:53:20.235263 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.235269    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"
Sep 10 23:53:20.235646 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.235296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr" event={"ID":"a24d016f-15ec-4aed-a8c3-dfe85091cc94","Type":"ContainerDied","Data":"00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445"}
Sep 10 23:53:20.235646 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.235336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr" event={"ID":"a24d016f-15ec-4aed-a8c3-dfe85091cc94","Type":"ContainerDied","Data":"4b91e49887c1b240cc1d87264f9a34e324d389463717af8c6c7dc15b38af6156"}
Sep 10 23:53:20.235646 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.235353    2578 scope.go:117] "RemoveContainer" containerID="00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445"
Sep 10 23:53:20.245256 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.245237    2578 scope.go:117] "RemoveContainer" containerID="00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445"
Sep 10 23:53:20.245578 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:53:20.245555    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445\": container with ID starting with 00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445 not found: ID does not exist" containerID="00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445"
Sep 10 23:53:20.245623 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.245587    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445"} err="failed to get container status \"00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445\": rpc error: code = NotFound desc = could not find container \"00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445\": container with ID starting with 00e4e8b25bf1acd4815036bac9d577ea23b11f0daf0fd37a42a4145d91913445 not found: ID does not exist"
Sep 10 23:53:20.257537 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.257509    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"]
Sep 10 23:53:20.259156 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.259138    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-ca420b8d-kserve-79544dcdd5-7wfqr"]
Sep 10 23:53:20.534205 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:20.534174    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24d016f-15ec-4aed-a8c3-dfe85091cc94" path="/var/lib/kubelet/pods/a24d016f-15ec-4aed-a8c3-dfe85091cc94/volumes"
Sep 10 23:53:23.198450 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:23.198409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:28.887194 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:28.887160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:28.991093 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:28.991052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:29.089638 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:29.089597    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:35.077433 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:35.077394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:35.332880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:35.332793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:35.801516 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:35.801483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:36.190157 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:36.190079    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:36.632475 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:36.632433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:38.278161 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:38.278125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:38.931982 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:38.931946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:39.025660 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:39.025626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:39.280778 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:39.280748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:39.650122 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:39.650039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:40.167103 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.167068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:40.289663 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.289621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:53:40.343688 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.343657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:40.952555 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.952520    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"]
Sep 10 23:53:40.952987 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.952968    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24d016f-15ec-4aed-a8c3-dfe85091cc94" containerName="main"
Sep 10 23:53:40.953096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.952989    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24d016f-15ec-4aed-a8c3-dfe85091cc94" containerName="main"
Sep 10 23:53:40.953096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.953020    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" containerName="maas-api"
Sep 10 23:53:40.953096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.953029    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" containerName="maas-api"
Sep 10 23:53:40.953096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.953059    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c75ca5-f97f-48ea-8aed-43c0119ef88e" containerName="main"
Sep 10 23:53:40.953096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.953067    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c75ca5-f97f-48ea-8aed-43c0119ef88e" containerName="main"
Sep 10 23:53:40.953386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.953160    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24d016f-15ec-4aed-a8c3-dfe85091cc94" containerName="main"
Sep 10 23:53:40.953386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.953175    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d1829b-1ac4-4c46-8da0-44c2a1bdd71a" containerName="maas-api"
Sep 10 23:53:40.953386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.953184    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c75ca5-f97f-48ea-8aed-43c0119ef88e" containerName="main"
Sep 10 23:53:40.956593 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.956555    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:40.959463 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.959439    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-7bb7cb-serving-cert\""
Sep 10 23:53:40.971130 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.971107    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"]
Sep 10 23:53:40.989210 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:40.989185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:41.090377 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.090346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:41.095428 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.095406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4kq\" (UniqueName: \"kubernetes.io/projected/26b001c0-2bd1-497a-ba62-3dc1f73667b7-kube-api-access-bx4kq\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.095519 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.095440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-maas-api-tls\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.095519 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.095486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b001c0-2bd1-497a-ba62-3dc1f73667b7-openshift-service-ca\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.095590 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.095533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-metrics-certs\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.196148 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.196121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-maas-api-tls\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.196258 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.196160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b001c0-2bd1-497a-ba62-3dc1f73667b7-openshift-service-ca\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.196258 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.196179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-metrics-certs\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.196258 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.196244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4kq\" (UniqueName: \"kubernetes.io/projected/26b001c0-2bd1-497a-ba62-3dc1f73667b7-kube-api-access-bx4kq\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.196887 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.196866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b001c0-2bd1-497a-ba62-3dc1f73667b7-openshift-service-ca\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.198650 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.198628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-maas-api-tls\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.198740 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.198658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-metrics-certs\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.206789 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.206735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4kq\" (UniqueName: \"kubernetes.io/projected/26b001c0-2bd1-497a-ba62-3dc1f73667b7-kube-api-access-bx4kq\") pod \"maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.248081 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.248046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:41.266810 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.266786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:41.546699 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.546664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"]
Sep 10 23:53:41.548265 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:53:41.548232    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b001c0_2bd1_497a_ba62_3dc1f73667b7.slice/crio-46f101f7e1d9e92871f53650c6a5c4fc4009ad68a80fe2291480abd5a17105e3 WatchSource:0}: Error finding container 46f101f7e1d9e92871f53650c6a5c4fc4009ad68a80fe2291480abd5a17105e3: Status 404 returned error can't find the container with id 46f101f7e1d9e92871f53650c6a5c4fc4009ad68a80fe2291480abd5a17105e3
Sep 10 23:53:41.549783 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.549760    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:53:41.636628 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.636596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:41.736248 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:41.736216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:42.350581 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:42.350544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6" event={"ID":"26b001c0-2bd1-497a-ba62-3dc1f73667b7","Type":"ContainerStarted","Data":"7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a"}
Sep 10 23:53:42.350747 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:42.350623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6" event={"ID":"26b001c0-2bd1-497a-ba62-3dc1f73667b7","Type":"ContainerStarted","Data":"46f101f7e1d9e92871f53650c6a5c4fc4009ad68a80fe2291480abd5a17105e3"}
Sep 10 23:53:42.350747 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:42.350646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:42.380120 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:42.380076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6" podStartSLOduration=1.9710504549999999 podStartE2EDuration="2.380061188s" totalImagesPullingTime="409.010733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:42.377796558 +0000 UTC m=+1064.334028362" watchObservedRunningTime="2026-09-10 23:53:42.380061188 +0000 UTC m=+1064.336292991"
Sep 10 23:53:46.199858 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:46.199826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:46.270289 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:46.270245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:46.372535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:46.372496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:46.470817 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:46.470742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:47.157245 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:47.157215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:47.253260 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:47.253228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:48.361202 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:48.361167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:53:48.879984 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:48.879949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:54.964889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:54.964802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:55.053028 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:55.052994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:55.505921 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:55.505883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:53:55.652043 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:55.652006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:59.943435 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:53:59.943396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:00.016012 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.015975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:00.249089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.249003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"]
Sep 10 23:54:00.254125 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.254102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.256894 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.256866    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-948e64\"/\"kube-root-ca.crt\""
Sep 10 23:54:00.257044 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.257028    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-948e64\"/\"openshift-service-ca.crt\""
Sep 10 23:54:00.258259 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.258239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-948e64\"/\"default-dockercfg-4bxrc\""
Sep 10 23:54:00.258433 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.258289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-948e64\"/\"test-model-948e64-kserve-self-signed-certs\""
Sep 10 23:54:00.264164 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.264141    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"]
Sep 10 23:54:00.266657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.266631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:00.375070 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.375039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv26z\" (UniqueName: \"kubernetes.io/projected/21b60546-2a3a-4eac-a428-d54b95a66f86-kube-api-access-cv26z\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.375243 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.375079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-model-cache\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.375243 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.375138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-tmp-dir\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.375243 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.375166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b60546-2a3a-4eac-a428-d54b95a66f86-tls-certs\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.375243 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.375207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-home\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.375400 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.375261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-dshm\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476174 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-dshm\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv26z\" (UniqueName: \"kubernetes.io/projected/21b60546-2a3a-4eac-a428-d54b95a66f86-kube-api-access-cv26z\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-model-cache\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-tmp-dir\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b60546-2a3a-4eac-a428-d54b95a66f86-tls-certs\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476607 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-home\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476740 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-model-cache\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476830 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-home\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.476877 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.476831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-tmp-dir\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.478528 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.478506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-dshm\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.478816 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.478797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b60546-2a3a-4eac-a428-d54b95a66f86-tls-certs\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.484993 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.484964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv26z\" (UniqueName: \"kubernetes.io/projected/21b60546-2a3a-4eac-a428-d54b95a66f86-kube-api-access-cv26z\") pod \"test-model-948e64-kserve-755dcdcc7c-ql5lb\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") " pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.564264 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.564225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:00.837719 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:00.837693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"]
Sep 10 23:54:00.839498 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:54:00.839472    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b60546_2a3a_4eac_a428_d54b95a66f86.slice/crio-778c08be8d282aabb7eca7d9f51eb7ef1d6e867874e1c65dce49b1f7e2b51022 WatchSource:0}: Error finding container 778c08be8d282aabb7eca7d9f51eb7ef1d6e867874e1c65dce49b1f7e2b51022: Status 404 returned error can't find the container with id 778c08be8d282aabb7eca7d9f51eb7ef1d6e867874e1c65dce49b1f7e2b51022
Sep 10 23:54:01.447469 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:01.447431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb" event={"ID":"21b60546-2a3a-4eac-a428-d54b95a66f86","Type":"ContainerStarted","Data":"aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8"}
Sep 10 23:54:01.447469 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:01.447470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb" event={"ID":"21b60546-2a3a-4eac-a428-d54b95a66f86","Type":"ContainerStarted","Data":"778c08be8d282aabb7eca7d9f51eb7ef1d6e867874e1c65dce49b1f7e2b51022"}
Sep 10 23:54:01.465372 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:01.465325    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb" podStartSLOduration=1.465311496 podStartE2EDuration="1.465311496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:01.464334618 +0000 UTC m=+1083.420566421" watchObservedRunningTime="2026-09-10 23:54:01.465311496 +0000 UTC m=+1083.421543298"
Sep 10 23:54:01.865854 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:01.865808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:01.966481 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:01.966448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:02.118426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:02.118338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:02.405348 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:02.405253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:02.542397 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:02.542358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:02.653591 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:02.653555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:03.368888 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:03.368855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:03.850096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:03.850063    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:04.471763 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:04.471730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:05.380036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:05.379997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:05.765249 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:05.765215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:06.019635 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:06.019553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:06.685422 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:06.685386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:07.717343 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:07.717312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:07.816029 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:07.815991    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:08.195686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:08.195652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:08.323084 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:08.323031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:08.426294 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:08.426248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:08.904294 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:08.904237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:09.739124 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:09.739089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:10.554408 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:10.554377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:10.565333 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:10.565267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:10.565461 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:10.565373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:10.577900 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:10.577874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:11.508080 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:11.508051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:54:15.247775 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:15.247741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:15.292307 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:15.292267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:54:15.349289 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:15.349240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:30.241548 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:30.241513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:30.610458 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:30.610416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.024346 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:31.024298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.233882 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:31.233840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.332772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:31.332677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.636508 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:31.636434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:32.076268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:32.076230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:32.992143 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:32.992108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:33.869733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:33.869696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:34.006076 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.006046    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:54:34.556962 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.556923    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"]
Sep 10 23:54:34.560575 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.560557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.565623 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.565601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7bb7cb\"/\"explicit-model-7bb7cb-kserve-self-signed-certs\""
Sep 10 23:54:34.566477 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.566456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7bb7cb\"/\"kube-root-ca.crt\""
Sep 10 23:54:34.566597 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.566507    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7bb7cb\"/\"default-dockercfg-5wlwf\""
Sep 10 23:54:34.566597 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.566510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7bb7cb\"/\"openshift-service-ca.crt\""
Sep 10 23:54:34.596532 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.596506    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"]
Sep 10 23:54:34.640112 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.640058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:34.685429 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.685398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4kq\" (UniqueName: \"kubernetes.io/projected/f932caec-15e5-4321-a1ba-f9723dc39a1e-kube-api-access-xp4kq\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.685594 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.685440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-model-cache\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.685594 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.685492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-dshm\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.685594 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.685530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-tmp-dir\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.685594 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.685575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f932caec-15e5-4321-a1ba-f9723dc39a1e-tls-certs\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.685759 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.685611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-home\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786177 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-home\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786410 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4kq\" (UniqueName: \"kubernetes.io/projected/f932caec-15e5-4321-a1ba-f9723dc39a1e-kube-api-access-xp4kq\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786410 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-model-cache\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786410 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-dshm\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786410 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-tmp-dir\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786627 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f932caec-15e5-4321-a1ba-f9723dc39a1e-tls-certs\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786627 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-home\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786746 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-model-cache\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.786746 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.786664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-tmp-dir\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.788618 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.788595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-dshm\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.788769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.788752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f932caec-15e5-4321-a1ba-f9723dc39a1e-tls-certs\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.808774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.808714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4kq\" (UniqueName: \"kubernetes.io/projected/f932caec-15e5-4321-a1ba-f9723dc39a1e-kube-api-access-xp4kq\") pod \"explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") " pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:34.869971 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:34.869943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:35.147047 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:35.147020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"]
Sep 10 23:54:35.147699 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:54:35.147672    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf932caec_15e5_4321_a1ba_f9723dc39a1e.slice/crio-50b82f61d3c30a53b6e54bf48c23743420a5a6dc29aefacb9f468963a92b3fe9 WatchSource:0}: Error finding container 50b82f61d3c30a53b6e54bf48c23743420a5a6dc29aefacb9f468963a92b3fe9: Status 404 returned error can't find the container with id 50b82f61d3c30a53b6e54bf48c23743420a5a6dc29aefacb9f468963a92b3fe9
Sep 10 23:54:35.613624 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:35.613591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw" event={"ID":"f932caec-15e5-4321-a1ba-f9723dc39a1e","Type":"ContainerStarted","Data":"67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230"}
Sep 10 23:54:35.613809 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:35.613637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw" event={"ID":"f932caec-15e5-4321-a1ba-f9723dc39a1e","Type":"ContainerStarted","Data":"50b82f61d3c30a53b6e54bf48c23743420a5a6dc29aefacb9f468963a92b3fe9"}
Sep 10 23:54:35.632341 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:35.632285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw" podStartSLOduration=1.6322550759999999 podStartE2EDuration="1.632255076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:35.631609208 +0000 UTC m=+1117.587841036" watchObservedRunningTime="2026-09-10 23:54:35.632255076 +0000 UTC m=+1117.588486880"
Sep 10 23:54:37.256426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:37.256391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:37.323248 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:37.323220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:38.763938 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:38.763906    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"]
Sep 10 23:54:38.828612 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:38.828570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:38.928527 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:38.928487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:40.092186 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:40.092151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:41.040040 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:41.040000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:41.749158 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:41.749124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:42.521813 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:42.521779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:43.346373 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:43.346335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:44.870549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:44.870515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:47.783559 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:47.783522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:48.443376 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:48.443318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:48.545380 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:48.545340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:48.642228 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:48.642189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:48.730360 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:48.730264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:49.011153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:49.011083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:49.119588 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:49.119552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:49.599933 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:49.599898    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:54:49.990055 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:49.989966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:50.661495 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.661461    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"]
Sep 10 23:54:50.661916 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.661704    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6" podUID="26b001c0-2bd1-497a-ba62-3dc1f73667b7" containerName="maas-api" containerID="cri-o://7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a" gracePeriod=30
Sep 10 23:54:50.695356 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.695325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:50.780119 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.780084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:50.868530 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.868482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:50.910685 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.910665    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:54:50.934870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.934814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-metrics-certs\") pod \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") "
Sep 10 23:54:50.934966 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.934944    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b001c0-2bd1-497a-ba62-3dc1f73667b7-openshift-service-ca\") pod \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") "
Sep 10 23:54:50.935007 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.934983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-maas-api-tls\") pod \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") "
Sep 10 23:54:50.935045 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.935011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx4kq\" (UniqueName: \"kubernetes.io/projected/26b001c0-2bd1-497a-ba62-3dc1f73667b7-kube-api-access-bx4kq\") pod \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\" (UID: \"26b001c0-2bd1-497a-ba62-3dc1f73667b7\") "
Sep 10 23:54:50.935355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.935332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b001c0-2bd1-497a-ba62-3dc1f73667b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26b001c0-2bd1-497a-ba62-3dc1f73667b7" (UID: "26b001c0-2bd1-497a-ba62-3dc1f73667b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:50.936834 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.936814    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26b001c0-2bd1-497a-ba62-3dc1f73667b7" (UID: "26b001c0-2bd1-497a-ba62-3dc1f73667b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:50.936974 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.936953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26b001c0-2bd1-497a-ba62-3dc1f73667b7" (UID: "26b001c0-2bd1-497a-ba62-3dc1f73667b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:50.937147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.937127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b001c0-2bd1-497a-ba62-3dc1f73667b7-kube-api-access-bx4kq" (OuterVolumeSpecName: "kube-api-access-bx4kq") pod "26b001c0-2bd1-497a-ba62-3dc1f73667b7" (UID: "26b001c0-2bd1-497a-ba62-3dc1f73667b7"). InnerVolumeSpecName "kube-api-access-bx4kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:50.943877 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:50.943847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:51.008073 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.008035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:51.036479 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.036450    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b001c0-2bd1-497a-ba62-3dc1f73667b7-openshift-service-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:51.036617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.036483    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-maas-api-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:51.036617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.036500    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4kq\" (UniqueName: \"kubernetes.io/projected/26b001c0-2bd1-497a-ba62-3dc1f73667b7-kube-api-access-bx4kq\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:51.036617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.036512    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b001c0-2bd1-497a-ba62-3dc1f73667b7-metrics-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:51.290585 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.290543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:51.697099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.697011    2578 generic.go:356] "Generic (PLEG): container finished" podID="26b001c0-2bd1-497a-ba62-3dc1f73667b7" containerID="7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a" exitCode=0
Sep 10 23:54:51.697099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.697050    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"
Sep 10 23:54:51.697099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.697094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6" event={"ID":"26b001c0-2bd1-497a-ba62-3dc1f73667b7","Type":"ContainerDied","Data":"7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a"}
Sep 10 23:54:51.697654 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.697142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6" event={"ID":"26b001c0-2bd1-497a-ba62-3dc1f73667b7","Type":"ContainerDied","Data":"46f101f7e1d9e92871f53650c6a5c4fc4009ad68a80fe2291480abd5a17105e3"}
Sep 10 23:54:51.697654 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.697169    2578 scope.go:117] "RemoveContainer" containerID="7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a"
Sep 10 23:54:51.706765 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.706749    2578 scope.go:117] "RemoveContainer" containerID="7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a"
Sep 10 23:54:51.707011 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:54:51.706994    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a\": container with ID starting with 7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a not found: ID does not exist" containerID="7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a"
Sep 10 23:54:51.707066 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.707017    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a"} err="failed to get container status \"7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a\": rpc error: code = NotFound desc = could not find container \"7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a\": container with ID starting with 7ee27ba16c6da79d6c10f1d117a5b10ab38e8c204ee80b7ffca367bbce49187a not found: ID does not exist"
Sep 10 23:54:51.734159 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.734130    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"]
Sep 10 23:54:51.758332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:51.758306    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7bb7cb-5d85dcc9bf-vc5m6"]
Sep 10 23:54:52.217656 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:52.217618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:52.534605 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:52.534563    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b001c0-2bd1-497a-ba62-3dc1f73667b7" path="/var/lib/kubelet/pods/26b001c0-2bd1-497a-ba62-3dc1f73667b7/volumes"
Sep 10 23:54:52.897740 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:52.897647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:53.608900 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:53.608865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:53.782903 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:53.782856    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw" podUID="f932caec-15e5-4321-a1ba-f9723dc39a1e" containerName="main" containerID="cri-o://67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230" gracePeriod=45
Sep 10 23:54:54.043930 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.043902    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:54.065656 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.065627    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-tmp-dir\") pod \"f932caec-15e5-4321-a1ba-f9723dc39a1e\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") "
Sep 10 23:54:54.065815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.065690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp4kq\" (UniqueName: \"kubernetes.io/projected/f932caec-15e5-4321-a1ba-f9723dc39a1e-kube-api-access-xp4kq\") pod \"f932caec-15e5-4321-a1ba-f9723dc39a1e\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") "
Sep 10 23:54:54.065815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.065716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f932caec-15e5-4321-a1ba-f9723dc39a1e-tls-certs\") pod \"f932caec-15e5-4321-a1ba-f9723dc39a1e\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") "
Sep 10 23:54:54.065815 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.065764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-model-cache\") pod \"f932caec-15e5-4321-a1ba-f9723dc39a1e\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") "
Sep 10 23:54:54.065977 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.065819    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-dshm\") pod \"f932caec-15e5-4321-a1ba-f9723dc39a1e\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") "
Sep 10 23:54:54.065977 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.065841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-home\") pod \"f932caec-15e5-4321-a1ba-f9723dc39a1e\" (UID: \"f932caec-15e5-4321-a1ba-f9723dc39a1e\") "
Sep 10 23:54:54.065977 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.065903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f932caec-15e5-4321-a1ba-f9723dc39a1e" (UID: "f932caec-15e5-4321-a1ba-f9723dc39a1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:54.066126 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.066105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f932caec-15e5-4321-a1ba-f9723dc39a1e" (UID: "f932caec-15e5-4321-a1ba-f9723dc39a1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:54.066126 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.066115    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:54.066352 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.066326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-home" (OuterVolumeSpecName: "home") pod "f932caec-15e5-4321-a1ba-f9723dc39a1e" (UID: "f932caec-15e5-4321-a1ba-f9723dc39a1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:54.068435 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.068397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-dshm" (OuterVolumeSpecName: "dshm") pod "f932caec-15e5-4321-a1ba-f9723dc39a1e" (UID: "f932caec-15e5-4321-a1ba-f9723dc39a1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:54.068543 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.068434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f932caec-15e5-4321-a1ba-f9723dc39a1e-kube-api-access-xp4kq" (OuterVolumeSpecName: "kube-api-access-xp4kq") pod "f932caec-15e5-4321-a1ba-f9723dc39a1e" (UID: "f932caec-15e5-4321-a1ba-f9723dc39a1e"). InnerVolumeSpecName "kube-api-access-xp4kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:54.068747 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.068718    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f932caec-15e5-4321-a1ba-f9723dc39a1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f932caec-15e5-4321-a1ba-f9723dc39a1e" (UID: "f932caec-15e5-4321-a1ba-f9723dc39a1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:54.166764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.166680    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4kq\" (UniqueName: \"kubernetes.io/projected/f932caec-15e5-4321-a1ba-f9723dc39a1e-kube-api-access-xp4kq\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:54.166764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.166711    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f932caec-15e5-4321-a1ba-f9723dc39a1e-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:54.166764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.166724    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:54.166764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.166731    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:54.166764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.166739    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f932caec-15e5-4321-a1ba-f9723dc39a1e-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:54:54.713882 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.713843    2578 generic.go:356] "Generic (PLEG): container finished" podID="f932caec-15e5-4321-a1ba-f9723dc39a1e" containerID="67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230" exitCode=0
Sep 10 23:54:54.713882 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.713882    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"
Sep 10 23:54:54.714097 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.713895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw" event={"ID":"f932caec-15e5-4321-a1ba-f9723dc39a1e","Type":"ContainerDied","Data":"67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230"}
Sep 10 23:54:54.714097 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.713930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw" event={"ID":"f932caec-15e5-4321-a1ba-f9723dc39a1e","Type":"ContainerDied","Data":"50b82f61d3c30a53b6e54bf48c23743420a5a6dc29aefacb9f468963a92b3fe9"}
Sep 10 23:54:54.714097 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.713948    2578 scope.go:117] "RemoveContainer" containerID="67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230"
Sep 10 23:54:54.722456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.722438    2578 scope.go:117] "RemoveContainer" containerID="67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230"
Sep 10 23:54:54.722714 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:54:54.722695    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230\": container with ID starting with 67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230 not found: ID does not exist" containerID="67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230"
Sep 10 23:54:54.722758 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.722723    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230"} err="failed to get container status \"67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230\": rpc error: code = NotFound desc = could not find container \"67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230\": container with ID starting with 67e7243109a6dfd32d6214e0f9d45705a6d9128e0a344c2392a4dfbf8901c230 not found: ID does not exist"
Sep 10 23:54:54.730430 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.730391    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"]
Sep 10 23:54:54.732471 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:54.732450    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7bb7cb/explicit-model-7bb7cb-kserve-66df6cc49f-8k9xw"]
Sep 10 23:54:55.133585 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:55.133550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:55.234520 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:55.234484    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:55.379867 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:55.379837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:55.720968 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:55.720933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:55.819884 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:55.819852    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:55.922135 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:55.922096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:56.533831 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:56.533800    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f932caec-15e5-4321-a1ba-f9723dc39a1e" path="/var/lib/kubelet/pods/f932caec-15e5-4321-a1ba-f9723dc39a1e/volumes"
Sep 10 23:54:57.102556 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:57.102520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:57.227287 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:57.227233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:57.414088 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:57.414005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:57.836071 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:57.836021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:58.176337 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:58.176234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:58.503444 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:58.503364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:59.663095 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:59.663059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:59.764569 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:59.764530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:59.867662 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:54:59.867625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:02.008029 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:02.007991    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:02.072426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:02.072362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:02.159946 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:02.159906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:02.257269 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:02.257237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:02.358543 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:02.358510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:05.099806 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:05.099773    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:55:08.056623 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:08.056579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:08.161154 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:08.161116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:08.990986 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:08.990945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:09.304441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:09.304398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:09.520554 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:09.520519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:11.666910 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:11.666865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:12.306036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:12.305987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:12.405796 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:12.405761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:12.645030 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:12.644944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:13.386710 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:13.386671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:14.174269 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.174230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:14.277485 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.277426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:14.333691 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.333652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:14.648284 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.648211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:14.731568 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.731531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:14.787183 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.787152    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"]
Sep 10 23:55:14.787608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.787583    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f932caec-15e5-4321-a1ba-f9723dc39a1e" containerName="main"
Sep 10 23:55:14.787608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.787611    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f932caec-15e5-4321-a1ba-f9723dc39a1e" containerName="main"
Sep 10 23:55:14.787701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.787635    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b001c0-2bd1-497a-ba62-3dc1f73667b7" containerName="maas-api"
Sep 10 23:55:14.787701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.787640    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b001c0-2bd1-497a-ba62-3dc1f73667b7" containerName="maas-api"
Sep 10 23:55:14.787780 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.787709    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b001c0-2bd1-497a-ba62-3dc1f73667b7" containerName="maas-api"
Sep 10 23:55:14.787780 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.787730    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f932caec-15e5-4321-a1ba-f9723dc39a1e" containerName="main"
Sep 10 23:55:14.792309 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.792284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:14.795128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.795101    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-396319\""
Sep 10 23:55:14.795128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.795121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-396319-dockercfg-xbs2x\""
Sep 10 23:55:14.800734 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.800704    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"]
Sep 10 23:55:14.859533 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.859498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68tx\" (UniqueName: \"kubernetes.io/projected/859f5875-2761-4211-b221-9a91ab9d770f-kube-api-access-b68tx\") pod \"payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:14.859688 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.859618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859f5875-2761-4211-b221-9a91ab9d770f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:14.960892 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.960783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859f5875-2761-4211-b221-9a91ab9d770f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:14.960892 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.960854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68tx\" (UniqueName: \"kubernetes.io/projected/859f5875-2761-4211-b221-9a91ab9d770f-kube-api-access-b68tx\") pod \"payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:14.961623 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.961596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859f5875-2761-4211-b221-9a91ab9d770f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:14.970197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:14.970167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68tx\" (UniqueName: \"kubernetes.io/projected/859f5875-2761-4211-b221-9a91ab9d770f-kube-api-access-b68tx\") pod \"payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:15.104428 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:15.104381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:15.378044 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:15.378019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"]
Sep 10 23:55:15.380066 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:55:15.380037    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859f5875_2761_4211_b221_9a91ab9d770f.slice/crio-68a287c6ec0ebf878fdfdd540d0e8a7287e23d753efc660735da1117078c1654 WatchSource:0}: Error finding container 68a287c6ec0ebf878fdfdd540d0e8a7287e23d753efc660735da1117078c1654: Status 404 returned error can't find the container with id 68a287c6ec0ebf878fdfdd540d0e8a7287e23d753efc660735da1117078c1654
Sep 10 23:55:15.817934 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:15.817893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7" event={"ID":"859f5875-2761-4211-b221-9a91ab9d770f","Type":"ContainerStarted","Data":"fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab"}
Sep 10 23:55:15.818335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:15.817954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7" event={"ID":"859f5875-2761-4211-b221-9a91ab9d770f","Type":"ContainerStarted","Data":"68a287c6ec0ebf878fdfdd540d0e8a7287e23d753efc660735da1117078c1654"}
Sep 10 23:55:15.818335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:15.817981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:15.846244 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:15.846198    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7" podStartSLOduration=1.8461823590000002 podStartE2EDuration="1.846182359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:15.844352717 +0000 UTC m=+1157.800584521" watchObservedRunningTime="2026-09-10 23:55:15.846182359 +0000 UTC m=+1157.802414214"
Sep 10 23:55:20.295756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:20.295705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:55:24.951025 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:24.950924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:26.825022 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:26.824994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 10 23:55:30.847302 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.847254    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"]
Sep 10 23:55:30.851404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.851379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:30.854206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.854186    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-76b16f\"/\"openshift-service-ca.crt\""
Sep 10 23:55:30.855565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.855539    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-76b16f\"/\"default-dockercfg-x6jzg\""
Sep 10 23:55:30.855680 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.855587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-76b16f\"/\"noten-model-76b16f-kserve-self-signed-certs\""
Sep 10 23:55:30.855878 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.855861    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-76b16f\"/\"kube-root-ca.crt\""
Sep 10 23:55:30.860709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.860686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"]
Sep 10 23:55:30.885873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.885845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:30.909343 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.909318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-dshm\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:30.909461 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.909355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tls-certs\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:30.909461 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.909373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-model-cache\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:30.909461 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.909446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-home\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:30.909572 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.909485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtqj\" (UniqueName: \"kubernetes.io/projected/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-kube-api-access-vqtqj\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:30.909572 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:30.909513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tmp-dir\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.009960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.009927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-dshm\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010105 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.009972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tls-certs\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010165 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.010144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-model-cache\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010228 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.010214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-home\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010315 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.010297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtqj\" (UniqueName: \"kubernetes.io/projected/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-kube-api-access-vqtqj\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010374 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.010351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tmp-dir\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.010541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-model-cache\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010701 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.010580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-home\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.010768 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.010746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tmp-dir\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.012146 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.012127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-dshm\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.012571 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.012552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tls-certs\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.018471 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.018447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtqj\" (UniqueName: \"kubernetes.io/projected/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-kube-api-access-vqtqj\") pod \"noten-model-76b16f-kserve-75cd9cd9cf-d4kzp\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") " pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.161833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.161754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:31.433186 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.433154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"]
Sep 10 23:55:31.435153 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:55:31.435122    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff0f6e0_4bb1_4a25_b017_4ead447a5c9c.slice/crio-d98ff7a9633c4c1b756a67b8d61f610589c1230cf01609fe0ff259d55876b640 WatchSource:0}: Error finding container d98ff7a9633c4c1b756a67b8d61f610589c1230cf01609fe0ff259d55876b640: Status 404 returned error can't find the container with id d98ff7a9633c4c1b756a67b8d61f610589c1230cf01609fe0ff259d55876b640
Sep 10 23:55:31.895825 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.895785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp" event={"ID":"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c","Type":"ContainerStarted","Data":"da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b"}
Sep 10 23:55:31.895825 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.895829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp" event={"ID":"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c","Type":"ContainerStarted","Data":"d98ff7a9633c4c1b756a67b8d61f610589c1230cf01609fe0ff259d55876b640"}
Sep 10 23:55:31.914741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:31.914695    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp" podStartSLOduration=1.914680595 podStartE2EDuration="1.914680595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:31.912988412 +0000 UTC m=+1173.869220228" watchObservedRunningTime="2026-09-10 23:55:31.914680595 +0000 UTC m=+1173.870912398"
Sep 10 23:55:33.225195 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:33.225154    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"]
Sep 10 23:55:33.244111 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:33.244080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:34.237944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:34.237907    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"]
Sep 10 23:55:34.248431 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:34.248392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:34.380301 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:34.380253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:34.623630 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:34.623593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:34.854179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:34.854136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:35.211856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:35.211820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:35.352535 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:35.352499    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"]
Sep 10 23:55:35.364165 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:35.364129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:36.163549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:36.163510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:36.873207 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:36.873170    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:37.278662 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:37.278630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:37.635371 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:37.635298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:37.734655 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:37.734616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:38.019464 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:38.019426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:38.162041 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:38.162001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:38.302601 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:38.302522    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:55:38.742327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:38.742228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:39.534537 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:39.534504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:39.823036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:39.822946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:40.701994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:40.701959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:41.162538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:41.162506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:41.649745 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:41.649710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:41.784654 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:41.784620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:42.668629 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:42.668594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:43.516342 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:43.516307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:44.209057 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:44.209028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:44.311171 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:44.311140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:44.361948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:44.361918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:44.868421 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:44.868389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:44.966521 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:44.966484    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:45.656612 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.656574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:45.689518 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.689486    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"]
Sep 10 23:55:45.689819 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.689793    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g" podUID="ed46c29a-db00-4cbf-81f6-bd97cad198b0" containerName="payload-processing" containerID="cri-o://58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27" gracePeriod=30
Sep 10 23:55:45.721775 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.721748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:45.821990 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.821942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:45.883450 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.883415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:45.944206 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.944180    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:55:45.972798 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.972763    2578 generic.go:356] "Generic (PLEG): container finished" podID="ed46c29a-db00-4cbf-81f6-bd97cad198b0" containerID="58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27" exitCode=0
Sep 10 23:55:45.972941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.972812    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"
Sep 10 23:55:45.972941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.972810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g" event={"ID":"ed46c29a-db00-4cbf-81f6-bd97cad198b0","Type":"ContainerDied","Data":"58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27"}
Sep 10 23:55:45.973016 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.972941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g" event={"ID":"ed46c29a-db00-4cbf-81f6-bd97cad198b0","Type":"ContainerDied","Data":"66d0ca9518dbd3e7a7c2c187152b893688d8d7d9005046eb9bb0159f1839c8dc"}
Sep 10 23:55:45.973016 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.972969    2578 scope.go:117] "RemoveContainer" containerID="58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27"
Sep 10 23:55:45.982482 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.982464    2578 scope.go:117] "RemoveContainer" containerID="58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27"
Sep 10 23:55:45.982767 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:55:45.982743    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27\": container with ID starting with 58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27 not found: ID does not exist" containerID="58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27"
Sep 10 23:55:45.982873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:45.982773    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27"} err="failed to get container status \"58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27\": rpc error: code = NotFound desc = could not find container \"58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27\": container with ID starting with 58056ffec995335a3590db8c319b0882eef87173bcb4f74d3b659b5516c1fc27 not found: ID does not exist"
Sep 10 23:55:46.059596 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.059565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed46c29a-db00-4cbf-81f6-bd97cad198b0-plugins-config-volume\") pod \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") "
Sep 10 23:55:46.059760 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.059622    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wznh4\" (UniqueName: \"kubernetes.io/projected/ed46c29a-db00-4cbf-81f6-bd97cad198b0-kube-api-access-wznh4\") pod \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\" (UID: \"ed46c29a-db00-4cbf-81f6-bd97cad198b0\") "
Sep 10 23:55:46.059980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.059957    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed46c29a-db00-4cbf-81f6-bd97cad198b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed46c29a-db00-4cbf-81f6-bd97cad198b0" (UID: "ed46c29a-db00-4cbf-81f6-bd97cad198b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:55:46.061651 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.061625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed46c29a-db00-4cbf-81f6-bd97cad198b0-kube-api-access-wznh4" (OuterVolumeSpecName: "kube-api-access-wznh4") pod "ed46c29a-db00-4cbf-81f6-bd97cad198b0" (UID: "ed46c29a-db00-4cbf-81f6-bd97cad198b0"). InnerVolumeSpecName "kube-api-access-wznh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:46.161000 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.160966    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed46c29a-db00-4cbf-81f6-bd97cad198b0-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:46.161000 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.160996    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznh4\" (UniqueName: \"kubernetes.io/projected/ed46c29a-db00-4cbf-81f6-bd97cad198b0-kube-api-access-wznh4\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:46.216249 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.216176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:46.296618 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.296582    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"]
Sep 10 23:55:46.298289 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.298251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:46.298632 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.298612    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c5699-845cd8d97c-j7d6g"]
Sep 10 23:55:46.534770 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.534735    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed46c29a-db00-4cbf-81f6-bd97cad198b0" path="/var/lib/kubelet/pods/ed46c29a-db00-4cbf-81f6-bd97cad198b0/volumes"
Sep 10 23:55:46.967617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:46.967531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:47.877732 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:47.877700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:48.244954 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.244865    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld" podUID="a3e5e664-e007-462f-987b-c9820e1da1a1" containerName="main" containerID="cri-o://fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd" gracePeriod=45
Sep 10 23:55:48.300438 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.300414    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:55:48.497715 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.497657    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:55:48.582021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.581990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f94r\" (UniqueName: \"kubernetes.io/projected/a3e5e664-e007-462f-987b-c9820e1da1a1-kube-api-access-5f94r\") pod \"a3e5e664-e007-462f-987b-c9820e1da1a1\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") "
Sep 10 23:55:48.582182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-tmp-dir\") pod \"a3e5e664-e007-462f-987b-c9820e1da1a1\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") "
Sep 10 23:55:48.582182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-home\") pod \"a3e5e664-e007-462f-987b-c9820e1da1a1\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") "
Sep 10 23:55:48.582182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-model-cache\") pod \"a3e5e664-e007-462f-987b-c9820e1da1a1\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") "
Sep 10 23:55:48.582182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-dshm\") pod \"a3e5e664-e007-462f-987b-c9820e1da1a1\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") "
Sep 10 23:55:48.582426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e664-e007-462f-987b-c9820e1da1a1-tls-certs\") pod \"a3e5e664-e007-462f-987b-c9820e1da1a1\" (UID: \"a3e5e664-e007-462f-987b-c9820e1da1a1\") "
Sep 10 23:55:48.582426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-home" (OuterVolumeSpecName: "home") pod "a3e5e664-e007-462f-987b-c9820e1da1a1" (UID: "a3e5e664-e007-462f-987b-c9820e1da1a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:48.582426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582411    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3e5e664-e007-462f-987b-c9820e1da1a1" (UID: "a3e5e664-e007-462f-987b-c9820e1da1a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:48.582549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3e5e664-e007-462f-987b-c9820e1da1a1" (UID: "a3e5e664-e007-462f-987b-c9820e1da1a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:48.582549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582532    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:48.582549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582544    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:48.582698 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.582552    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:48.584293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.584239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e5e664-e007-462f-987b-c9820e1da1a1-kube-api-access-5f94r" (OuterVolumeSpecName: "kube-api-access-5f94r") pod "a3e5e664-e007-462f-987b-c9820e1da1a1" (UID: "a3e5e664-e007-462f-987b-c9820e1da1a1"). InnerVolumeSpecName "kube-api-access-5f94r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:48.584293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.584284    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-dshm" (OuterVolumeSpecName: "dshm") pod "a3e5e664-e007-462f-987b-c9820e1da1a1" (UID: "a3e5e664-e007-462f-987b-c9820e1da1a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:48.584447 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.584268    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e5e664-e007-462f-987b-c9820e1da1a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3e5e664-e007-462f-987b-c9820e1da1a1" (UID: "a3e5e664-e007-462f-987b-c9820e1da1a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:55:48.683522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.683497    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e664-e007-462f-987b-c9820e1da1a1-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:48.683522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.683518    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e664-e007-462f-987b-c9820e1da1a1-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:48.683680 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.683530    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f94r\" (UniqueName: \"kubernetes.io/projected/a3e5e664-e007-462f-987b-c9820e1da1a1-kube-api-access-5f94r\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:48.712141 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.712111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:48.988079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.988043    2578 generic.go:356] "Generic (PLEG): container finished" podID="a3e5e664-e007-462f-987b-c9820e1da1a1" containerID="fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd" exitCode=0
Sep 10 23:55:48.988255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.988083    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"
Sep 10 23:55:48.988255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.988130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld" event={"ID":"a3e5e664-e007-462f-987b-c9820e1da1a1","Type":"ContainerDied","Data":"fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd"}
Sep 10 23:55:48.988255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.988165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld" event={"ID":"a3e5e664-e007-462f-987b-c9820e1da1a1","Type":"ContainerDied","Data":"327fbd11fc575f0745ae2138d675d251004720f2f6f0be3ad95fc40dbb69acf9"}
Sep 10 23:55:48.988255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.988183    2578 scope.go:117] "RemoveContainer" containerID="fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd"
Sep 10 23:55:48.997102 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.997080    2578 scope.go:117] "RemoveContainer" containerID="fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd"
Sep 10 23:55:48.997389 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:55:48.997372    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd\": container with ID starting with fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd not found: ID does not exist" containerID="fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd"
Sep 10 23:55:48.997444 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:48.997397    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd"} err="failed to get container status \"fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd\": rpc error: code = NotFound desc = could not find container \"fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd\": container with ID starting with fb47db3dd408e472f4784d0ba9d1acc224142dc8529d6e3e7f62a611a91280bd not found: ID does not exist"
Sep 10 23:55:49.009714 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.009692    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"]
Sep 10 23:55:49.014090 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.014070    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4c5699/test-model-4c5699-kserve-5bd6c99d4f-lppld"]
Sep 10 23:55:49.257527 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.257422    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp" podUID="eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" containerName="main" containerID="cri-o://da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b" gracePeriod=45
Sep 10 23:55:49.513561 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.513542    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:49.590718 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.590687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqtqj\" (UniqueName: \"kubernetes.io/projected/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-kube-api-access-vqtqj\") pod \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") "
Sep 10 23:55:49.590870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.590785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-home\") pod \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") "
Sep 10 23:55:49.590870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.590811    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-dshm\") pod \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") "
Sep 10 23:55:49.590870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.590829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tmp-dir\") pod \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") "
Sep 10 23:55:49.591036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.590870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tls-certs\") pod \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") "
Sep 10 23:55:49.591036 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.590910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-model-cache\") pod \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\" (UID: \"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c\") "
Sep 10 23:55:49.591144 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.591074    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-home" (OuterVolumeSpecName: "home") pod "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" (UID: "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:49.591144 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.591104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" (UID: "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:49.591302 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.591255    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" (UID: "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:49.591439 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.591420    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:49.591502 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.591457    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:49.591502 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.591485    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:49.593128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.593101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-dshm" (OuterVolumeSpecName: "dshm") pod "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" (UID: "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:49.593229 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.593137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" (UID: "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:55:49.593229 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.593147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-kube-api-access-vqtqj" (OuterVolumeSpecName: "kube-api-access-vqtqj") pod "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" (UID: "eff0f6e0-4bb1-4a25-b017-4ead447a5c9c"). InnerVolumeSpecName "kube-api-access-vqtqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:49.692076 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.692049    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:49.692076 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.692072    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqtqj\" (UniqueName: \"kubernetes.io/projected/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-kube-api-access-vqtqj\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:49.692220 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.692082    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:49.995386 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.995304    2578 generic.go:356] "Generic (PLEG): container finished" podID="eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" containerID="da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b" exitCode=0
Sep 10 23:55:49.995525 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.995385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp" event={"ID":"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c","Type":"ContainerDied","Data":"da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b"}
Sep 10 23:55:49.995525 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.995416    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"
Sep 10 23:55:49.995525 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.995425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp" event={"ID":"eff0f6e0-4bb1-4a25-b017-4ead447a5c9c","Type":"ContainerDied","Data":"d98ff7a9633c4c1b756a67b8d61f610589c1230cf01609fe0ff259d55876b640"}
Sep 10 23:55:49.995525 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:49.995446    2578 scope.go:117] "RemoveContainer" containerID="da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b"
Sep 10 23:55:50.004344 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.004324    2578 scope.go:117] "RemoveContainer" containerID="da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b"
Sep 10 23:55:50.004640 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:55:50.004616    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b\": container with ID starting with da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b not found: ID does not exist" containerID="da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b"
Sep 10 23:55:50.004741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.004645    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b"} err="failed to get container status \"da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b\": rpc error: code = NotFound desc = could not find container \"da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b\": container with ID starting with da5f1b5270e032521cd35b742f8e6fa3ba58370cf4b5812e60e14cf87479b78b not found: ID does not exist"
Sep 10 23:55:50.016510 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.016483    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"]
Sep 10 23:55:50.018044 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.018021    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-76b16f/noten-model-76b16f-kserve-75cd9cd9cf-d4kzp"]
Sep 10 23:55:50.371190 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.371149    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb" podUID="21b60546-2a3a-4eac-a428-d54b95a66f86" containerName="main" containerID="cri-o://aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8" gracePeriod=45
Sep 10 23:55:50.535766 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.535556    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e5e664-e007-462f-987b-c9820e1da1a1" path="/var/lib/kubelet/pods/a3e5e664-e007-462f-987b-c9820e1da1a1/volumes"
Sep 10 23:55:50.536165 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.536140    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" path="/var/lib/kubelet/pods/eff0f6e0-4bb1-4a25-b017-4ead447a5c9c/volumes"
Sep 10 23:55:50.624581 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.624527    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:55:50.700671 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.700640    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-tmp-dir\") pod \"21b60546-2a3a-4eac-a428-d54b95a66f86\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") "
Sep 10 23:55:50.700803 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.700709    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv26z\" (UniqueName: \"kubernetes.io/projected/21b60546-2a3a-4eac-a428-d54b95a66f86-kube-api-access-cv26z\") pod \"21b60546-2a3a-4eac-a428-d54b95a66f86\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") "
Sep 10 23:55:50.700803 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.700746    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-model-cache\") pod \"21b60546-2a3a-4eac-a428-d54b95a66f86\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") "
Sep 10 23:55:50.700922 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.700829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b60546-2a3a-4eac-a428-d54b95a66f86-tls-certs\") pod \"21b60546-2a3a-4eac-a428-d54b95a66f86\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") "
Sep 10 23:55:50.700922 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.700857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-dshm\") pod \"21b60546-2a3a-4eac-a428-d54b95a66f86\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") "
Sep 10 23:55:50.700922 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.700884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-home\") pod \"21b60546-2a3a-4eac-a428-d54b95a66f86\" (UID: \"21b60546-2a3a-4eac-a428-d54b95a66f86\") "
Sep 10 23:55:50.700922 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.700916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21b60546-2a3a-4eac-a428-d54b95a66f86" (UID: "21b60546-2a3a-4eac-a428-d54b95a66f86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:50.701142 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.701078    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-model-cache" (OuterVolumeSpecName: "model-cache") pod "21b60546-2a3a-4eac-a428-d54b95a66f86" (UID: "21b60546-2a3a-4eac-a428-d54b95a66f86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:50.701197 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.701170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-home" (OuterVolumeSpecName: "home") pod "21b60546-2a3a-4eac-a428-d54b95a66f86" (UID: "21b60546-2a3a-4eac-a428-d54b95a66f86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:50.701260 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.701243    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:50.701346 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.701266    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:50.702935 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.702905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-dshm" (OuterVolumeSpecName: "dshm") pod "21b60546-2a3a-4eac-a428-d54b95a66f86" (UID: "21b60546-2a3a-4eac-a428-d54b95a66f86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:55:50.703047 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.702969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b60546-2a3a-4eac-a428-d54b95a66f86-kube-api-access-cv26z" (OuterVolumeSpecName: "kube-api-access-cv26z") pod "21b60546-2a3a-4eac-a428-d54b95a66f86" (UID: "21b60546-2a3a-4eac-a428-d54b95a66f86"). InnerVolumeSpecName "kube-api-access-cv26z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:50.703047 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.702968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b60546-2a3a-4eac-a428-d54b95a66f86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21b60546-2a3a-4eac-a428-d54b95a66f86" (UID: "21b60546-2a3a-4eac-a428-d54b95a66f86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:55:50.802661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.802631    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv26z\" (UniqueName: \"kubernetes.io/projected/21b60546-2a3a-4eac-a428-d54b95a66f86-kube-api-access-cv26z\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:50.802661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.802656    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b60546-2a3a-4eac-a428-d54b95a66f86-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:50.802661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.802667    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:50.802853 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:50.802675    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b60546-2a3a-4eac-a428-d54b95a66f86-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:55:51.003160 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.003078    2578 generic.go:356] "Generic (PLEG): container finished" podID="21b60546-2a3a-4eac-a428-d54b95a66f86" containerID="aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8" exitCode=0
Sep 10 23:55:51.003160 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.003121    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"
Sep 10 23:55:51.003160 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.003144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb" event={"ID":"21b60546-2a3a-4eac-a428-d54b95a66f86","Type":"ContainerDied","Data":"aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8"}
Sep 10 23:55:51.003458 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.003173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb" event={"ID":"21b60546-2a3a-4eac-a428-d54b95a66f86","Type":"ContainerDied","Data":"778c08be8d282aabb7eca7d9f51eb7ef1d6e867874e1c65dce49b1f7e2b51022"}
Sep 10 23:55:51.003458 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.003189    2578 scope.go:117] "RemoveContainer" containerID="aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8"
Sep 10 23:55:51.012533 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.012514    2578 scope.go:117] "RemoveContainer" containerID="aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8"
Sep 10 23:55:51.012804 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:55:51.012785    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8\": container with ID starting with aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8 not found: ID does not exist" containerID="aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8"
Sep 10 23:55:51.012860 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.012813    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8"} err="failed to get container status \"aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8\": rpc error: code = NotFound desc = could not find container \"aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8\": container with ID starting with aa50ec84edaf3833853a99407cf4ca8ae79aab540f61a149b7824c607dc62fd8 not found: ID does not exist"
Sep 10 23:55:51.024879 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.024854    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"]
Sep 10 23:55:51.028613 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:51.028594    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-948e64/test-model-948e64-kserve-755dcdcc7c-ql5lb"]
Sep 10 23:55:52.061592 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:52.061557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:52.162024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:52.161992    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:52.320244 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:52.320164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:52.534196 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:52.534162    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b60546-2a3a-4eac-a428-d54b95a66f86" path="/var/lib/kubelet/pods/21b60546-2a3a-4eac-a428-d54b95a66f86/volumes"
Sep 10 23:55:52.998778 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:52.998737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:53.900051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:53.900024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:54.652133 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:54.652096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:55.525488 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:55.525450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:56.357630 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:56.357592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:56.610822 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:56.610732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:57.132029 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:57.131994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:57.808805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:57.808770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:58.462747 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:58.462691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:58.580678 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:58.580653    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:55:58.581622 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:58.581598    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 10 23:55:59.177525 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:55:59.177494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:01.291503 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:01.291468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:02.052700 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.052668    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7744-ce161e"]
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053016    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e5e664-e007-462f-987b-c9820e1da1a1" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053028    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e5e664-e007-462f-987b-c9820e1da1a1" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053044    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b60546-2a3a-4eac-a428-d54b95a66f86" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053050    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b60546-2a3a-4eac-a428-d54b95a66f86" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053063    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed46c29a-db00-4cbf-81f6-bd97cad198b0" containerName="payload-processing"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053069    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed46c29a-db00-4cbf-81f6-bd97cad198b0" containerName="payload-processing"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053080    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053085    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053138    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff0f6e0-4bb1-4a25-b017-4ead447a5c9c" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053145    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed46c29a-db00-4cbf-81f6-bd97cad198b0" containerName="payload-processing"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053152    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e5e664-e007-462f-987b-c9820e1da1a1" containerName="main"
Sep 10 23:56:02.054832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.053158    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b60546-2a3a-4eac-a428-d54b95a66f86" containerName="main"
Sep 10 23:56:02.056013 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.055995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:02.058583 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.058563    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wtk84\""
Sep 10 23:56:02.062638 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.062616    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-ce161e"]
Sep 10 23:56:02.204310 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.204253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/abd4ab8a-a333-4754-8043-600b9b56c9fb-kube-api-access-zz257\") pod \"test-curl-7744-ce161e\" (UID: \"abd4ab8a-a333-4754-8043-600b9b56c9fb\") " pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:02.305772 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.305691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/abd4ab8a-a333-4754-8043-600b9b56c9fb-kube-api-access-zz257\") pod \"test-curl-7744-ce161e\" (UID: \"abd4ab8a-a333-4754-8043-600b9b56c9fb\") " pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:02.314714 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.314693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/abd4ab8a-a333-4754-8043-600b9b56c9fb-kube-api-access-zz257\") pod \"test-curl-7744-ce161e\" (UID: \"abd4ab8a-a333-4754-8043-600b9b56c9fb\") " pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:02.366944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.366920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:02.635080 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:02.635049    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-ce161e"]
Sep 10 23:56:02.636414 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:02.636386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabd4ab8a_a333_4754_8043_600b9b56c9fb.slice/crio-0a738e9cfb74792af294a24d715b060006b5d2d653c333fcbc803c2d5f7952b5 WatchSource:0}: Error finding container 0a738e9cfb74792af294a24d715b060006b5d2d653c333fcbc803c2d5f7952b5: Status 404 returned error can't find the container with id 0a738e9cfb74792af294a24d715b060006b5d2d653c333fcbc803c2d5f7952b5
Sep 10 23:56:03.064072 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.064036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-ce161e" event={"ID":"abd4ab8a-a333-4754-8043-600b9b56c9fb","Type":"ContainerStarted","Data":"f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2"}
Sep 10 23:56:03.064072 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.064084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-ce161e" event={"ID":"abd4ab8a-a333-4754-8043-600b9b56c9fb","Type":"ContainerStarted","Data":"0a738e9cfb74792af294a24d715b060006b5d2d653c333fcbc803c2d5f7952b5"}
Sep 10 23:56:03.080126 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.080071    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7744-ce161e" podStartSLOduration=1.080053935 podStartE2EDuration="1.080053935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:03.078342231 +0000 UTC m=+1205.034574035" watchObservedRunningTime="2026-09-10 23:56:03.080053935 +0000 UTC m=+1205.036285738"
Sep 10 23:56:03.180203 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.180168    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"]
Sep 10 23:56:03.180453 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.180423    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc" podUID="33cb34c5-8674-4568-bf00-238a5b5b9059" containerName="maas-api" containerID="cri-o://8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec" gracePeriod=30
Sep 10 23:56:03.216058 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.216020    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:03.242379 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.242339    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"]
Sep 10 23:56:03.242625 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.242598    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp" podUID="555d01b6-5141-4a91-a144-804689cca34d" containerName="payload-processing" containerID="cri-o://1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5" gracePeriod=30
Sep 10 23:56:03.250145 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.250041    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"]
Sep 10 23:56:03.250319 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.250254    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b" podUID="5753bc76-f54b-4212-9964-aeb2302fa8b3" containerName="payload-pre-processing" containerID="cri-o://9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16" gracePeriod=30
Sep 10 23:56:03.314765 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.314694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:03.416737 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.416394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:03.485312 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.485258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:03.548946 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.548920    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:56:03.551995 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.551973    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:56:03.638842 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.638810    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:56:03.718610 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5753bc76-f54b-4212-9964-aeb2302fa8b3-plugins-config-volume\") pod \"5753bc76-f54b-4212-9964-aeb2302fa8b3\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") "
Sep 10 23:56:03.718769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndkql\" (UniqueName: \"kubernetes.io/projected/555d01b6-5141-4a91-a144-804689cca34d-kube-api-access-ndkql\") pod \"555d01b6-5141-4a91-a144-804689cca34d\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") "
Sep 10 23:56:03.718769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718669    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9c2w\" (UniqueName: \"kubernetes.io/projected/33cb34c5-8674-4568-bf00-238a5b5b9059-kube-api-access-n9c2w\") pod \"33cb34c5-8674-4568-bf00-238a5b5b9059\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") "
Sep 10 23:56:03.718769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718701    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cb34c5-8674-4568-bf00-238a5b5b9059-openshift-service-ca\") pod \"33cb34c5-8674-4568-bf00-238a5b5b9059\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") "
Sep 10 23:56:03.718994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-metrics-certs\") pod \"33cb34c5-8674-4568-bf00-238a5b5b9059\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") "
Sep 10 23:56:03.718994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr279\" (UniqueName: \"kubernetes.io/projected/5753bc76-f54b-4212-9964-aeb2302fa8b3-kube-api-access-kr279\") pod \"5753bc76-f54b-4212-9964-aeb2302fa8b3\" (UID: \"5753bc76-f54b-4212-9964-aeb2302fa8b3\") "
Sep 10 23:56:03.718994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls\") pod \"33cb34c5-8674-4568-bf00-238a5b5b9059\" (UID: \"33cb34c5-8674-4568-bf00-238a5b5b9059\") "
Sep 10 23:56:03.718994 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.718980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555d01b6-5141-4a91-a144-804689cca34d-plugins-config-volume\") pod \"555d01b6-5141-4a91-a144-804689cca34d\" (UID: \"555d01b6-5141-4a91-a144-804689cca34d\") "
Sep 10 23:56:03.719193 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.719038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5753bc76-f54b-4212-9964-aeb2302fa8b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5753bc76-f54b-4212-9964-aeb2302fa8b3" (UID: "5753bc76-f54b-4212-9964-aeb2302fa8b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:56:03.719362 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.719332    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5753bc76-f54b-4212-9964-aeb2302fa8b3-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.719473 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.719436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cb34c5-8674-4568-bf00-238a5b5b9059-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33cb34c5-8674-4568-bf00-238a5b5b9059" (UID: "33cb34c5-8674-4568-bf00-238a5b5b9059"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:56:03.719547 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.719500    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555d01b6-5141-4a91-a144-804689cca34d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "555d01b6-5141-4a91-a144-804689cca34d" (UID: "555d01b6-5141-4a91-a144-804689cca34d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:56:03.720941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.720912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cb34c5-8674-4568-bf00-238a5b5b9059-kube-api-access-n9c2w" (OuterVolumeSpecName: "kube-api-access-n9c2w") pod "33cb34c5-8674-4568-bf00-238a5b5b9059" (UID: "33cb34c5-8674-4568-bf00-238a5b5b9059"). InnerVolumeSpecName "kube-api-access-n9c2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:56:03.721049 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.720985    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555d01b6-5141-4a91-a144-804689cca34d-kube-api-access-ndkql" (OuterVolumeSpecName: "kube-api-access-ndkql") pod "555d01b6-5141-4a91-a144-804689cca34d" (UID: "555d01b6-5141-4a91-a144-804689cca34d"). InnerVolumeSpecName "kube-api-access-ndkql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:56:03.721049 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.721020    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33cb34c5-8674-4568-bf00-238a5b5b9059" (UID: "33cb34c5-8674-4568-bf00-238a5b5b9059"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:56:03.721452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.721429    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33cb34c5-8674-4568-bf00-238a5b5b9059" (UID: "33cb34c5-8674-4568-bf00-238a5b5b9059"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:56:03.721542 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.721457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5753bc76-f54b-4212-9964-aeb2302fa8b3-kube-api-access-kr279" (OuterVolumeSpecName: "kube-api-access-kr279") pod "5753bc76-f54b-4212-9964-aeb2302fa8b3" (UID: "5753bc76-f54b-4212-9964-aeb2302fa8b3"). InnerVolumeSpecName "kube-api-access-kr279". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:56:03.732357 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.732243    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7744-ce161e"]
Sep 10 23:56:03.739617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.739595    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7744-ce161e"]
Sep 10 23:56:03.820156 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.820128    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-metrics-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.820156 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.820154    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr279\" (UniqueName: \"kubernetes.io/projected/5753bc76-f54b-4212-9964-aeb2302fa8b3-kube-api-access-kr279\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.820355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.820164    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33cb34c5-8674-4568-bf00-238a5b5b9059-maas-api-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.820355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.820173    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555d01b6-5141-4a91-a144-804689cca34d-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.820355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.820194    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndkql\" (UniqueName: \"kubernetes.io/projected/555d01b6-5141-4a91-a144-804689cca34d-kube-api-access-ndkql\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.820355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.820205    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9c2w\" (UniqueName: \"kubernetes.io/projected/33cb34c5-8674-4568-bf00-238a5b5b9059-kube-api-access-n9c2w\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.820355 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.820213    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cb34c5-8674-4568-bf00-238a5b5b9059-openshift-service-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:03.828139 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.828113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:03.857179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857158    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7744-cf15ee"]
Sep 10 23:56:03.857514 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857501    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555d01b6-5141-4a91-a144-804689cca34d" containerName="payload-processing"
Sep 10 23:56:03.857563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="555d01b6-5141-4a91-a144-804689cca34d" containerName="payload-processing"
Sep 10 23:56:03.857563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857530    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cb34c5-8674-4568-bf00-238a5b5b9059" containerName="maas-api"
Sep 10 23:56:03.857563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857535    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cb34c5-8674-4568-bf00-238a5b5b9059" containerName="maas-api"
Sep 10 23:56:03.857563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857543    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd4ab8a-a333-4754-8043-600b9b56c9fb" containerName="test-curl-7744-ce161e"
Sep 10 23:56:03.857563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857548    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd4ab8a-a333-4754-8043-600b9b56c9fb" containerName="test-curl-7744-ce161e"
Sep 10 23:56:03.857563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857560    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5753bc76-f54b-4212-9964-aeb2302fa8b3" containerName="payload-pre-processing"
Sep 10 23:56:03.857563 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857565    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5753bc76-f54b-4212-9964-aeb2302fa8b3" containerName="payload-pre-processing"
Sep 10 23:56:03.857774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857617    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="555d01b6-5141-4a91-a144-804689cca34d" containerName="payload-processing"
Sep 10 23:56:03.857774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857625    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5753bc76-f54b-4212-9964-aeb2302fa8b3" containerName="payload-pre-processing"
Sep 10 23:56:03.857774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857633    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd4ab8a-a333-4754-8043-600b9b56c9fb" containerName="test-curl-7744-ce161e"
Sep 10 23:56:03.857774 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.857638    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33cb34c5-8674-4568-bf00-238a5b5b9059" containerName="maas-api"
Sep 10 23:56:03.861630 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.861614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:03.870329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.870301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-cf15ee"]
Sep 10 23:56:03.993961 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:03.993928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:04.021705 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.021673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgh24\" (UniqueName: \"kubernetes.io/projected/c0e45fa2-1879-4341-83d6-bae7b2331c9e-kube-api-access-bgh24\") pod \"test-curl-7744-cf15ee\" (UID: \"c0e45fa2-1879-4341-83d6-bae7b2331c9e\") " pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:04.069856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.069825    2578 generic.go:356] "Generic (PLEG): container finished" podID="555d01b6-5141-4a91-a144-804689cca34d" containerID="1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5" exitCode=0
Sep 10 23:56:04.069987 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.069880    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"
Sep 10 23:56:04.069987 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.069911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp" event={"ID":"555d01b6-5141-4a91-a144-804689cca34d","Type":"ContainerDied","Data":"1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5"}
Sep 10 23:56:04.070079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.069990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp" event={"ID":"555d01b6-5141-4a91-a144-804689cca34d","Type":"ContainerDied","Data":"2328768a328eaaf02efa5c7217c39fb5d307d42ce371880cecb64782881a3b7e"}
Sep 10 23:56:04.070079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.070019    2578 scope.go:117] "RemoveContainer" containerID="1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5"
Sep 10 23:56:04.071383 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.071360    2578 generic.go:356] "Generic (PLEG): container finished" podID="33cb34c5-8674-4568-bf00-238a5b5b9059" containerID="8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec" exitCode=0
Sep 10 23:56:04.071491 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.071388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc" event={"ID":"33cb34c5-8674-4568-bf00-238a5b5b9059","Type":"ContainerDied","Data":"8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec"}
Sep 10 23:56:04.071491 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.071417    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"
Sep 10 23:56:04.071491 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.071444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc" event={"ID":"33cb34c5-8674-4568-bf00-238a5b5b9059","Type":"ContainerDied","Data":"ed2bc45fe530feae466355f1d15dd70371593ae723400c60386e17b2f3a06844"}
Sep 10 23:56:04.072856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.072823    2578 generic.go:356] "Generic (PLEG): container finished" podID="5753bc76-f54b-4212-9964-aeb2302fa8b3" containerID="9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16" exitCode=0
Sep 10 23:56:04.072957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.072879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"
Sep 10 23:56:04.072957 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.072909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b" event={"ID":"5753bc76-f54b-4212-9964-aeb2302fa8b3","Type":"ContainerDied","Data":"9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16"}
Sep 10 23:56:04.073047 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.072957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b" event={"ID":"5753bc76-f54b-4212-9964-aeb2302fa8b3","Type":"ContainerDied","Data":"845cee4ad4685f574081e63e7d059b8d1d78b7d10c407c545a8d89881578d3f9"}
Sep 10 23:56:04.073187 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:04.073170    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7744-ce161e" container="test-curl-7744-ce161e"
Sep 10 23:56:04.080976 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.080959    2578 scope.go:117] "RemoveContainer" containerID="1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5"
Sep 10 23:56:04.081258 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:04.081235    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5\": container with ID starting with 1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5 not found: ID does not exist" containerID="1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5"
Sep 10 23:56:04.081384 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.081264    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5"} err="failed to get container status \"1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5\": rpc error: code = NotFound desc = could not find container \"1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5\": container with ID starting with 1ced49ea46f682c9587dca51086365125c07cab2216229e98a3cf56a2538c6d5 not found: ID does not exist"
Sep 10 23:56:04.081384 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.081323    2578 scope.go:117] "RemoveContainer" containerID="8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec"
Sep 10 23:56:04.089487 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.089471    2578 scope.go:117] "RemoveContainer" containerID="8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec"
Sep 10 23:56:04.089737 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:04.089719    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec\": container with ID starting with 8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec not found: ID does not exist" containerID="8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec"
Sep 10 23:56:04.089812 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.089746    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec"} err="failed to get container status \"8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec\": rpc error: code = NotFound desc = could not find container \"8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec\": container with ID starting with 8e4ecbf8aef1b6eef0a9471021f25ec4b34f417d1bc4a2bee316ecc041f0fcec not found: ID does not exist"
Sep 10 23:56:04.089812 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.089769    2578 scope.go:117] "RemoveContainer" containerID="9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16"
Sep 10 23:56:04.097304 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.097287    2578 scope.go:117] "RemoveContainer" containerID="9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16"
Sep 10 23:56:04.097591 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:04.097575    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16\": container with ID starting with 9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16 not found: ID does not exist" containerID="9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16"
Sep 10 23:56:04.097656 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.097600    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16"} err="failed to get container status \"9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16\": rpc error: code = NotFound desc = could not find container \"9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16\": container with ID starting with 9952a48109728d7c7fdbae934c3f41efbc18ceec4d1dca83245ef4a7c4acbf16 not found: ID does not exist"
Sep 10 23:56:04.098456 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.098436    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"]
Sep 10 23:56:04.104600 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.104576    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948e64-67b67f49cf-4p7rp"]
Sep 10 23:56:04.118112 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.118091    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"]
Sep 10 23:56:04.122466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.122409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgh24\" (UniqueName: \"kubernetes.io/projected/c0e45fa2-1879-4341-83d6-bae7b2331c9e-kube-api-access-bgh24\") pod \"test-curl-7744-cf15ee\" (UID: \"c0e45fa2-1879-4341-83d6-bae7b2331c9e\") " pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:04.122760 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.122740    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-948e64-78bd8896c8-5njcc"]
Sep 10 23:56:04.132734 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.132712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgh24\" (UniqueName: \"kubernetes.io/projected/c0e45fa2-1879-4341-83d6-bae7b2331c9e-kube-api-access-bgh24\") pod \"test-curl-7744-cf15ee\" (UID: \"c0e45fa2-1879-4341-83d6-bae7b2331c9e\") " pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:04.136031 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.136009    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"]
Sep 10 23:56:04.140895 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.140873    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948e64-d6777c9c8-ftj9b"]
Sep 10 23:56:04.173684 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.173666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:04.450174 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.450148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-cf15ee"]
Sep 10 23:56:04.451635 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:04.451600    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e45fa2_1879_4341_83d6_bae7b2331c9e.slice/crio-b3dc5c05ed691bec435b4dd85e20d150b8e10bb5edb6c094c45eea8f853bbe52 WatchSource:0}: Error finding container b3dc5c05ed691bec435b4dd85e20d150b8e10bb5edb6c094c45eea8f853bbe52: Status 404 returned error can't find the container with id b3dc5c05ed691bec435b4dd85e20d150b8e10bb5edb6c094c45eea8f853bbe52
Sep 10 23:56:04.534021 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.533995    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33cb34c5-8674-4568-bf00-238a5b5b9059" path="/var/lib/kubelet/pods/33cb34c5-8674-4568-bf00-238a5b5b9059/volumes"
Sep 10 23:56:04.534419 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.534405    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555d01b6-5141-4a91-a144-804689cca34d" path="/var/lib/kubelet/pods/555d01b6-5141-4a91-a144-804689cca34d/volumes"
Sep 10 23:56:04.534757 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:04.534741    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5753bc76-f54b-4212-9964-aeb2302fa8b3" path="/var/lib/kubelet/pods/5753bc76-f54b-4212-9964-aeb2302fa8b3/volumes"
Sep 10 23:56:05.040520 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.040468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:05.081374 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.081341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-cf15ee" event={"ID":"c0e45fa2-1879-4341-83d6-bae7b2331c9e","Type":"ContainerStarted","Data":"5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4"}
Sep 10 23:56:05.081554 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.081382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-cf15ee" event={"ID":"c0e45fa2-1879-4341-83d6-bae7b2331c9e","Type":"ContainerStarted","Data":"b3dc5c05ed691bec435b4dd85e20d150b8e10bb5edb6c094c45eea8f853bbe52"}
Sep 10 23:56:05.081554 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.081394    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7744-ce161e" podUID="abd4ab8a-a333-4754-8043-600b9b56c9fb" containerName="test-curl-7744-ce161e" containerID="cri-o://f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2" gracePeriod=2
Sep 10 23:56:05.103908 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.103864    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7744-cf15ee" podStartSLOduration=2.10384984 podStartE2EDuration="2.10384984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:05.10261763 +0000 UTC m=+1207.058849434" watchObservedRunningTime="2026-09-10 23:56:05.10384984 +0000 UTC m=+1207.060081643"
Sep 10 23:56:05.734145 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.734064    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7744-cf15ee"]
Sep 10 23:56:05.741094 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.741064    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7744-cf15ee"]
Sep 10 23:56:05.854248 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.854215    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7744-339c88"]
Sep 10 23:56:05.854609 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.854596    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e45fa2-1879-4341-83d6-bae7b2331c9e" containerName="test-curl-7744-cf15ee"
Sep 10 23:56:05.854656 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.854611    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e45fa2-1879-4341-83d6-bae7b2331c9e" containerName="test-curl-7744-cf15ee"
Sep 10 23:56:05.854713 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.854703    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e45fa2-1879-4341-83d6-bae7b2331c9e" containerName="test-curl-7744-cf15ee"
Sep 10 23:56:05.858932 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.858916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:05.864527 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.864501    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-339c88"]
Sep 10 23:56:05.878122 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.878098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:05.939471 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:05.939430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9jg\" (UniqueName: \"kubernetes.io/projected/2452c3ee-f958-4c31-a02d-d16449d56cdb-kube-api-access-7c9jg\") pod \"test-curl-7744-339c88\" (UID: \"2452c3ee-f958-4c31-a02d-d16449d56cdb\") " pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:06.040868 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:06.040822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9jg\" (UniqueName: \"kubernetes.io/projected/2452c3ee-f958-4c31-a02d-d16449d56cdb-kube-api-access-7c9jg\") pod \"test-curl-7744-339c88\" (UID: \"2452c3ee-f958-4c31-a02d-d16449d56cdb\") " pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:06.048849 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:06.048822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9jg\" (UniqueName: \"kubernetes.io/projected/2452c3ee-f958-4c31-a02d-d16449d56cdb-kube-api-access-7c9jg\") pod \"test-curl-7744-339c88\" (UID: \"2452c3ee-f958-4c31-a02d-d16449d56cdb\") " pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:06.085942 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:06.085916    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7744-cf15ee" container="test-curl-7744-cf15ee"
Sep 10 23:56:06.170089 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:06.170057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:06.437106 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:06.437079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-339c88"]
Sep 10 23:56:06.439049 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:06.439018    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2452c3ee_f958_4c31_a02d_d16449d56cdb.slice/crio-650bea030b313c815ac37803caab2baf21c66475cbaf5803291fa4ac39607cc5 WatchSource:0}: Error finding container 650bea030b313c815ac37803caab2baf21c66475cbaf5803291fa4ac39607cc5: Status 404 returned error can't find the container with id 650bea030b313c815ac37803caab2baf21c66475cbaf5803291fa4ac39607cc5
Sep 10 23:56:06.590583 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:06.590542    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:56:07.093024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.092987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-339c88" event={"ID":"2452c3ee-f958-4c31-a02d-d16449d56cdb","Type":"ContainerStarted","Data":"e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de"}
Sep 10 23:56:07.093481 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.093036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-339c88" event={"ID":"2452c3ee-f958-4c31-a02d-d16449d56cdb","Type":"ContainerStarted","Data":"650bea030b313c815ac37803caab2baf21c66475cbaf5803291fa4ac39607cc5"}
Sep 10 23:56:07.093481 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.093062    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7744-cf15ee" podUID="c0e45fa2-1879-4341-83d6-bae7b2331c9e" containerName="test-curl-7744-cf15ee" containerID="cri-o://5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4" gracePeriod=2
Sep 10 23:56:07.108314 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.108245    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7744-339c88" podStartSLOduration=2.108228453 podStartE2EDuration="2.108228453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:07.106391761 +0000 UTC m=+1209.062623565" watchObservedRunningTime="2026-09-10 23:56:07.108228453 +0000 UTC m=+1209.064460255"
Sep 10 23:56:07.256109 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.256083    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:07.352454 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.352370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/abd4ab8a-a333-4754-8043-600b9b56c9fb-kube-api-access-zz257\") pod \"abd4ab8a-a333-4754-8043-600b9b56c9fb\" (UID: \"abd4ab8a-a333-4754-8043-600b9b56c9fb\") "
Sep 10 23:56:07.354579 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.354545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd4ab8a-a333-4754-8043-600b9b56c9fb-kube-api-access-zz257" (OuterVolumeSpecName: "kube-api-access-zz257") pod "abd4ab8a-a333-4754-8043-600b9b56c9fb" (UID: "abd4ab8a-a333-4754-8043-600b9b56c9fb"). InnerVolumeSpecName "kube-api-access-zz257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:56:07.453876 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.453847    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/abd4ab8a-a333-4754-8043-600b9b56c9fb-kube-api-access-zz257\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:07.663810 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.663731    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7744-339c88"]
Sep 10 23:56:07.669163 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.669128    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7744-339c88"]
Sep 10 23:56:07.783191 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.783161    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7744-e30c28"]
Sep 10 23:56:07.783557 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.783544    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2452c3ee-f958-4c31-a02d-d16449d56cdb" containerName="test-curl-7744-339c88"
Sep 10 23:56:07.783609 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.783558    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2452c3ee-f958-4c31-a02d-d16449d56cdb" containerName="test-curl-7744-339c88"
Sep 10 23:56:07.783643 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.783617    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2452c3ee-f958-4c31-a02d-d16449d56cdb" containerName="test-curl-7744-339c88"
Sep 10 23:56:07.786599 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.786583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:07.790735 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.790708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-e30c28"]
Sep 10 23:56:07.959127 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:07.959058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qsb\" (UniqueName: \"kubernetes.io/projected/0440ee81-4230-4567-9d79-4f86dffe373b-kube-api-access-p9qsb\") pod \"test-curl-7744-e30c28\" (UID: \"0440ee81-4230-4567-9d79-4f86dffe373b\") " pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:08.060390 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.060360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qsb\" (UniqueName: \"kubernetes.io/projected/0440ee81-4230-4567-9d79-4f86dffe373b-kube-api-access-p9qsb\") pod \"test-curl-7744-e30c28\" (UID: \"0440ee81-4230-4567-9d79-4f86dffe373b\") " pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:08.073578 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.073553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qsb\" (UniqueName: \"kubernetes.io/projected/0440ee81-4230-4567-9d79-4f86dffe373b-kube-api-access-p9qsb\") pod \"test-curl-7744-e30c28\" (UID: \"0440ee81-4230-4567-9d79-4f86dffe373b\") " pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:08.097157 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.097132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:08.099518 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.099487    2578 generic.go:356] "Generic (PLEG): container finished" podID="abd4ab8a-a333-4754-8043-600b9b56c9fb" containerID="f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2" exitCode=137
Sep 10 23:56:08.099637 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.099534    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:08.099637 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.099586    2578 scope.go:117] "RemoveContainer" containerID="f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2"
Sep 10 23:56:08.099881 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:08.099862    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7744-339c88" container="test-curl-7744-339c88"
Sep 10 23:56:08.109907 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.109884    2578 scope.go:117] "RemoveContainer" containerID="f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2"
Sep 10 23:56:08.110222 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:08.110202    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2\": container with ID starting with f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2 not found: ID does not exist" containerID="f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2"
Sep 10 23:56:08.110329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.110231    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2"} err="failed to get container status \"f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2\": rpc error: code = NotFound desc = could not find container \"f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2\": container with ID starting with f136c1a0b38f21c20096e3feb5b0e0a6f9abd573ff31d0b6f89f00afc846d3a2 not found: ID does not exist"
Sep 10 23:56:08.126183 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:08.126151    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7744-ce161e\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="abd4ab8a-a333-4754-8043-600b9b56c9fb" pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:08.371824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.371794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7744-e30c28"]
Sep 10 23:56:08.375218 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:08.375183    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0440ee81_4230_4567_9d79_4f86dffe373b.slice/crio-eb02b08e0bcee97eca19f6e592e319f95ea84e2f50ff86e4bd2dd617b6931deb WatchSource:0}: Error finding container eb02b08e0bcee97eca19f6e592e319f95ea84e2f50ff86e4bd2dd617b6931deb: Status 404 returned error can't find the container with id eb02b08e0bcee97eca19f6e592e319f95ea84e2f50ff86e4bd2dd617b6931deb
Sep 10 23:56:08.531934 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:08.531902    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7744-ce161e\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="abd4ab8a-a333-4754-8043-600b9b56c9fb" pod="opendatahub/test-curl-7744-ce161e"
Sep 10 23:56:08.534091 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:08.534064    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd4ab8a-a333-4754-8043-600b9b56c9fb" path="/var/lib/kubelet/pods/abd4ab8a-a333-4754-8043-600b9b56c9fb/volumes"
Sep 10 23:56:09.106554 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.106525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-e30c28" event={"ID":"0440ee81-4230-4567-9d79-4f86dffe373b","Type":"ContainerStarted","Data":"f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620"}
Sep 10 23:56:09.106953 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.106569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7744-e30c28" event={"ID":"0440ee81-4230-4567-9d79-4f86dffe373b","Type":"ContainerStarted","Data":"eb02b08e0bcee97eca19f6e592e319f95ea84e2f50ff86e4bd2dd617b6931deb"}
Sep 10 23:56:09.106953 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.106610    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7744-339c88" podUID="2452c3ee-f958-4c31-a02d-d16449d56cdb" containerName="test-curl-7744-339c88" containerID="cri-o://e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de" gracePeriod=2
Sep 10 23:56:09.120452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.120387    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7744-e30c28" podStartSLOduration=2.120370379 podStartE2EDuration="2.120370379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:09.120350155 +0000 UTC m=+1211.076581961" watchObservedRunningTime="2026-09-10 23:56:09.120370379 +0000 UTC m=+1211.076602182"
Sep 10 23:56:09.266118 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.266095    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:09.372656 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.372579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgh24\" (UniqueName: \"kubernetes.io/projected/c0e45fa2-1879-4341-83d6-bae7b2331c9e-kube-api-access-bgh24\") pod \"c0e45fa2-1879-4341-83d6-bae7b2331c9e\" (UID: \"c0e45fa2-1879-4341-83d6-bae7b2331c9e\") "
Sep 10 23:56:09.374347 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.374326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e45fa2-1879-4341-83d6-bae7b2331c9e-kube-api-access-bgh24" (OuterVolumeSpecName: "kube-api-access-bgh24") pod "c0e45fa2-1879-4341-83d6-bae7b2331c9e" (UID: "c0e45fa2-1879-4341-83d6-bae7b2331c9e"). InnerVolumeSpecName "kube-api-access-bgh24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:56:09.474105 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.474076    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgh24\" (UniqueName: \"kubernetes.io/projected/c0e45fa2-1879-4341-83d6-bae7b2331c9e-kube-api-access-bgh24\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:09.635311 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.635225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:09.703915 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.703882    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7744-e30c28"]
Sep 10 23:56:09.709560 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.709529    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7744-e30c28"]
Sep 10 23:56:09.734419 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.734381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:09.975296 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:09.975192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:10.112335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.112304    2578 generic.go:356] "Generic (PLEG): container finished" podID="c0e45fa2-1879-4341-83d6-bae7b2331c9e" containerID="5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4" exitCode=137
Sep 10 23:56:10.112764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.112348    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:10.112764 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.112404    2578 scope.go:117] "RemoveContainer" containerID="5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4"
Sep 10 23:56:10.112764 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:10.112692    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7744-e30c28" container="test-curl-7744-e30c28"
Sep 10 23:56:10.120844 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.120787    2578 scope.go:117] "RemoveContainer" containerID="5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4"
Sep 10 23:56:10.121055 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:10.121039    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4\": container with ID starting with 5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4 not found: ID does not exist" containerID="5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4"
Sep 10 23:56:10.121102 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.121063    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4"} err="failed to get container status \"5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4\": rpc error: code = NotFound desc = could not find container \"5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4\": container with ID starting with 5bceb7b790d1b67b5ae2138fd0db41969fefa35e5f289f96aef4d029f5ba3ad4 not found: ID does not exist"
Sep 10 23:56:10.122718 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:10.122698    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7744-cf15ee\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="c0e45fa2-1879-4341-83d6-bae7b2331c9e" pod="opendatahub/test-curl-7744-cf15ee"
Sep 10 23:56:10.299814 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.299769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:10.451988 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.451942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:10.534819 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.534781    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e45fa2-1879-4341-83d6-bae7b2331c9e" path="/var/lib/kubelet/pods/c0e45fa2-1879-4341-83d6-bae7b2331c9e/volumes"
Sep 10 23:56:10.563639 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.563568    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"]
Sep 10 23:56:10.563938 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.563925    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0440ee81-4230-4567-9d79-4f86dffe373b" containerName="test-curl-7744-e30c28"
Sep 10 23:56:10.563980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.563940    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0440ee81-4230-4567-9d79-4f86dffe373b" containerName="test-curl-7744-e30c28"
Sep 10 23:56:10.564013 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.563999    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0440ee81-4230-4567-9d79-4f86dffe373b" containerName="test-curl-7744-e30c28"
Sep 10 23:56:10.568216 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.568195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.570881 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.570859    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 23:56:10.570993 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.570976    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f8d9df10-openshift-default-dockercfg-dsnb4\""
Sep 10 23:56:10.583657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.583627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"]
Sep 10 23:56:10.590513 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.590477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:10.684655 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-credential-socket\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.684816 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-socket\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.684816 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-envoy\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.684816 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3842711-b605-4843-9039-bb0440effb50-istiod-ca-cert\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.684967 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxm4\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-kube-api-access-wxxm4\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.684967 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3842711-b605-4843-9039-bb0440effb50-istio-podinfo\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.684967 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-certs\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.685148 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.684977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-istio-token\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.685148 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.685062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-data\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.738457 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.738424    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"]
Sep 10 23:56:10.741944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.741909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.746010 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.745990    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6fa41e-openshift-default-dockercfg-tctlv\""
Sep 10 23:56:10.753332 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.753305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"]
Sep 10 23:56:10.784293 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.784239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:10.785862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.785838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-certs\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.785968 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.785877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-istio-token\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.785968 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.785903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-data\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.785968 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.785951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-credential-socket\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.785980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-socket\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.786005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-envoy\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.786052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3842711-b605-4843-9039-bb0440effb50-istiod-ca-cert\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786128 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.786104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxm4\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-kube-api-access-wxxm4\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786650 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.786468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3842711-b605-4843-9039-bb0440effb50-istio-podinfo\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.786661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-credential-socket\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.786673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-certs\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.786956 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.786918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-socket\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.787056 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.787031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-data\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.787974 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.787945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3842711-b605-4843-9039-bb0440effb50-istiod-ca-cert\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.795190 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.789860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3842711-b605-4843-9039-bb0440effb50-istio-podinfo\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.795735 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.795701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-envoy\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.795822 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.795759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-istio-token\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.806098 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.806071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxm4\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-kube-api-access-wxxm4\") pod \"e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") " pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.879986 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.879908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:10.886993 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.886957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:10.887379 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ede721-c87a-419c-89b0-3ef6dbbab219-istiod-ca-cert\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887442 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkrr\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-kube-api-access-tmkrr\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-data\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887516 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-certs\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887564 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-credential-socket\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887611 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-podinfo\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887668 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-token\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887709 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-socket\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.887755 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.887743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-envoy\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988220 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ede721-c87a-419c-89b0-3ef6dbbab219-istiod-ca-cert\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkrr\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-kube-api-access-tmkrr\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:10.988402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-data\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-certs\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-credential-socket\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988402 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-podinfo\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-token\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-socket\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-envoy\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-data\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988945 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-certs\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988995 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-socket\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988995 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ede721-c87a-419c-89b0-3ef6dbbab219-istiod-ca-cert\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.988995 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.988976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-credential-socket\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.990571 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.990549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-envoy\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.990829 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.990812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-podinfo\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.996762 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.996741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkrr\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-kube-api-access-tmkrr\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:10.997598 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:10.997581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-token\") pod \"e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") " pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:11.054639 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.054613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:11.121002 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.120971    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7744-e30c28" podUID="0440ee81-4230-4567-9d79-4f86dffe373b" containerName="test-curl-7744-e30c28" containerID="cri-o://f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620" gracePeriod=2
Sep 10 23:56:11.156635 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.156609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"]
Sep 10 23:56:11.159146 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:11.159114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3842711_b605_4843_9039_bb0440effb50.slice/crio-3d2ad331c73b9f73e732587882b6509cb8193cd9eeb89fdd5018be0a25a881af WatchSource:0}: Error finding container 3d2ad331c73b9f73e732587882b6509cb8193cd9eeb89fdd5018be0a25a881af: Status 404 returned error can't find the container with id 3d2ad331c73b9f73e732587882b6509cb8193cd9eeb89fdd5018be0a25a881af
Sep 10 23:56:11.248455 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.248426    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:11.326211 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.326186    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"]
Sep 10 23:56:11.327663 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:11.327632    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ede721_c87a_419c_89b0_3ef6dbbab219.slice/crio-a74f655b6745d98fe1e75806365895ecac21efe6b95f7e65626de18e0bad5d40 WatchSource:0}: Error finding container a74f655b6745d98fe1e75806365895ecac21efe6b95f7e65626de18e0bad5d40: Status 404 returned error can't find the container with id a74f655b6745d98fe1e75806365895ecac21efe6b95f7e65626de18e0bad5d40
Sep 10 23:56:11.392949 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.392881    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c9jg\" (UniqueName: \"kubernetes.io/projected/2452c3ee-f958-4c31-a02d-d16449d56cdb-kube-api-access-7c9jg\") pod \"2452c3ee-f958-4c31-a02d-d16449d56cdb\" (UID: \"2452c3ee-f958-4c31-a02d-d16449d56cdb\") "
Sep 10 23:56:11.394909 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.394879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2452c3ee-f958-4c31-a02d-d16449d56cdb-kube-api-access-7c9jg" (OuterVolumeSpecName: "kube-api-access-7c9jg") pod "2452c3ee-f958-4c31-a02d-d16449d56cdb" (UID: "2452c3ee-f958-4c31-a02d-d16449d56cdb"). InnerVolumeSpecName "kube-api-access-7c9jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:56:11.493840 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:11.493817    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c9jg\" (UniqueName: \"kubernetes.io/projected/2452c3ee-f958-4c31-a02d-d16449d56cdb-kube-api-access-7c9jg\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:12.126848 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.126757    2578 generic.go:356] "Generic (PLEG): container finished" podID="2452c3ee-f958-4c31-a02d-d16449d56cdb" containerID="e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de" exitCode=137
Sep 10 23:56:12.126848 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.126802    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:12.127382 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.126865    2578 scope.go:117] "RemoveContainer" containerID="e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de"
Sep 10 23:56:12.128183 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.128110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" event={"ID":"a3842711-b605-4843-9039-bb0440effb50","Type":"ContainerStarted","Data":"3d2ad331c73b9f73e732587882b6509cb8193cd9eeb89fdd5018be0a25a881af"}
Sep 10 23:56:12.129504 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.129480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" event={"ID":"30ede721-c87a-419c-89b0-3ef6dbbab219","Type":"ContainerStarted","Data":"a74f655b6745d98fe1e75806365895ecac21efe6b95f7e65626de18e0bad5d40"}
Sep 10 23:56:12.136312 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.136293    2578 scope.go:117] "RemoveContainer" containerID="e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de"
Sep 10 23:56:12.136579 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:12.136556    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de\": container with ID starting with e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de not found: ID does not exist" containerID="e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de"
Sep 10 23:56:12.136661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.136584    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de"} err="failed to get container status \"e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de\": rpc error: code = NotFound desc = could not find container \"e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de\": container with ID starting with e131c62c4a4c6fc8e1160383e4a6fbc2e60a9b9b5aff2a9067919c9a97e4e4de not found: ID does not exist"
Sep 10 23:56:12.138572 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:12.138543    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7744-339c88\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="2452c3ee-f958-4c31-a02d-d16449d56cdb" pod="opendatahub/test-curl-7744-339c88"
Sep 10 23:56:12.538579 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:12.538542    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2452c3ee-f958-4c31-a02d-d16449d56cdb" path="/var/lib/kubelet/pods/2452c3ee-f958-4c31-a02d-d16449d56cdb/volumes"
Sep 10 23:56:13.326722 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:13.326689    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:13.413078 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:13.412996    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9qsb\" (UniqueName: \"kubernetes.io/projected/0440ee81-4230-4567-9d79-4f86dffe373b-kube-api-access-p9qsb\") pod \"0440ee81-4230-4567-9d79-4f86dffe373b\" (UID: \"0440ee81-4230-4567-9d79-4f86dffe373b\") "
Sep 10 23:56:13.415428 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:13.415400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0440ee81-4230-4567-9d79-4f86dffe373b-kube-api-access-p9qsb" (OuterVolumeSpecName: "kube-api-access-p9qsb") pod "0440ee81-4230-4567-9d79-4f86dffe373b" (UID: "0440ee81-4230-4567-9d79-4f86dffe373b"). InnerVolumeSpecName "kube-api-access-p9qsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:56:13.513986 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:13.513949    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qsb\" (UniqueName: \"kubernetes.io/projected/0440ee81-4230-4567-9d79-4f86dffe373b-kube-api-access-p9qsb\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:56:14.145690 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.145651    2578 generic.go:356] "Generic (PLEG): container finished" podID="0440ee81-4230-4567-9d79-4f86dffe373b" containerID="f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620" exitCode=137
Sep 10 23:56:14.145880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.145703    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:14.145880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.145760    2578 scope.go:117] "RemoveContainer" containerID="f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620"
Sep 10 23:56:14.157560 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:14.157525    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7744-e30c28\" is forbidden: User \"system:node:ip-10-0-128-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-88.ec2.internal' and this object" podUID="0440ee81-4230-4567-9d79-4f86dffe373b" pod="opendatahub/test-curl-7744-e30c28"
Sep 10 23:56:14.445893 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.445652    2578 scope.go:117] "RemoveContainer" containerID="f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620"
Sep 10 23:56:14.446134 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:14.445999    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620\": container with ID starting with f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620 not found: ID does not exist" containerID="f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620"
Sep 10 23:56:14.446134 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.446027    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620"} err="failed to get container status \"f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620\": rpc error: code = NotFound desc = could not find container \"f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620\": container with ID starting with f4555d18afd39cd9e15553d62f77789605af25bbf7b08210254360613f9b4620 not found: ID does not exist"
Sep 10 23:56:14.521377 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.521326    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:14.521472 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.521428    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:14.521521 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.521471    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:14.527875 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.527775    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:14.527875 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.527860    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:14.527954 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.527909    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:14.533704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:14.533680    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0440ee81-4230-4567-9d79-4f86dffe373b" path="/var/lib/kubelet/pods/0440ee81-4230-4567-9d79-4f86dffe373b/volumes"
Sep 10 23:56:15.155264 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:15.155221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" event={"ID":"a3842711-b605-4843-9039-bb0440effb50","Type":"ContainerStarted","Data":"414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f"}
Sep 10 23:56:15.156941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:15.156912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" event={"ID":"30ede721-c87a-419c-89b0-3ef6dbbab219","Type":"ContainerStarted","Data":"4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80"}
Sep 10 23:56:15.177599 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:15.177552    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podStartSLOduration=1.817995706 podStartE2EDuration="5.177536439s" totalImagesPullingTime="3.359540733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:15.175204024 +0000 UTC m=+1217.131435828" watchObservedRunningTime="2026-09-10 23:56:15.177536439 +0000 UTC m=+1217.133768241"
Sep 10 23:56:15.195064 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:15.195012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" podStartSLOduration=1.997239901 podStartE2EDuration="5.194993789s" totalImagesPullingTime="3.197753888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:15.19302571 +0000 UTC m=+1217.149257513" watchObservedRunningTime="2026-09-10 23:56:15.194993789 +0000 UTC m=+1217.151225591"
Sep 10 23:56:15.881111 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:15.881070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:15.886207 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:15.886179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:16.054837 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.054794    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:16.059862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.059835    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:16.161771 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.161651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:16.161771 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.161714    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:16.162467 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.162443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:16.162581 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.162511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:16.290186 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.290147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:16.388335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.388303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:16.489404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:16.489305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:17.100325 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:17.100263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:17.228072 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:17.228034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:17.324225 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:17.324184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:17.614598 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:17.614543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:18.029193 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:18.029157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:18.581226 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:18.581185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:19.326778 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.326748    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"]
Sep 10 23:56:19.331316 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.331288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.334294 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.334229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:56:19.334294 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.334263    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-78nn9\""
Sep 10 23:56:19.334467 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.334229    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f8d9df10-serving-cert\""
Sep 10 23:56:19.335440 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.335422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:56:19.335554 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.335491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:56:19.343062 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.343039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"]
Sep 10 23:56:19.357364 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.357337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:19.372543 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.372508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tb7\" (UniqueName: \"kubernetes.io/projected/5c2c01a0-9311-4e14-8f82-09ea34e84b27-kube-api-access-92tb7\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.372686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.372563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2c01a0-9311-4e14-8f82-09ea34e84b27-openshift-service-ca\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.372686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.372653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.372686 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.372684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-metrics-certs\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.457675 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.457641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:19.473639 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.473610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-metrics-certs\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.473756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.473665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tb7\" (UniqueName: \"kubernetes.io/projected/5c2c01a0-9311-4e14-8f82-09ea34e84b27-kube-api-access-92tb7\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.473756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.473736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2c01a0-9311-4e14-8f82-09ea34e84b27-openshift-service-ca\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.473827 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.473812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.473928 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:19.473911    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-serving-cert: secret "maas-api-e2e-mt-f8d9df10-serving-cert" not found
Sep 10 23:56:19.473994 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:56:19.473984    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls podName:5c2c01a0-9311-4e14-8f82-09ea34e84b27 nodeName:}" failed. No retries permitted until 2026-09-10 23:56:19.973963096 +0000 UTC m=+1221.930194892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls") pod "maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" (UID: "5c2c01a0-9311-4e14-8f82-09ea34e84b27") : secret "maas-api-e2e-mt-f8d9df10-serving-cert" not found
Sep 10 23:56:19.474395 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.474374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2c01a0-9311-4e14-8f82-09ea34e84b27-openshift-service-ca\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.476052 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.476032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-metrics-certs\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.484298 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.484251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tb7\" (UniqueName: \"kubernetes.io/projected/5c2c01a0-9311-4e14-8f82-09ea34e84b27-kube-api-access-92tb7\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.596668 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.596593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:19.824099 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.824065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:19.926985 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.926901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:19.932797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.932733    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"]
Sep 10 23:56:19.936549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.936525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:19.939258 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.939231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8d9df10-dockercfg-vg795\""
Sep 10 23:56:19.939860 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.939836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8d9df10\""
Sep 10 23:56:19.946960 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.946937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"]
Sep 10 23:56:19.977657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.977631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05595a49-13ac-4995-ba71-5c21513add9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:19.977770 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.977696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:19.977846 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.977770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl82k\" (UniqueName: \"kubernetes.io/projected/05595a49-13ac-4995-ba71-5c21513add9f-kube-api-access-bl82k\") pod \"payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:19.979932 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:19.979915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls\") pod \"maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:20.079135 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.079099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl82k\" (UniqueName: \"kubernetes.io/projected/05595a49-13ac-4995-ba71-5c21513add9f-kube-api-access-bl82k\") pod \"payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:20.079399 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.079172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05595a49-13ac-4995-ba71-5c21513add9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:20.079870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.079848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05595a49-13ac-4995-ba71-5c21513add9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:20.089356 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.089336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl82k\" (UniqueName: \"kubernetes.io/projected/05595a49-13ac-4995-ba71-5c21513add9f-kube-api-access-bl82k\") pod \"payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:20.238247 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.238166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:20.242801 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.242777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:20.247837 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.247819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:20.355941 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.355906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:20.511493 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.511417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:20.520566 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.520543    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"]
Sep 10 23:56:20.522885 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:20.522791    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2c01a0_9311_4e14_8f82_09ea34e84b27.slice/crio-b951f174ba7efdb5c3d07af35cc9dfdf24649e89145bcd17aa30c66605a61007 WatchSource:0}: Error finding container b951f174ba7efdb5c3d07af35cc9dfdf24649e89145bcd17aa30c66605a61007: Status 404 returned error can't find the container with id b951f174ba7efdb5c3d07af35cc9dfdf24649e89145bcd17aa30c66605a61007
Sep 10 23:56:20.524778 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.524674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"]
Sep 10 23:56:20.526368 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:20.526339    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05595a49_13ac_4995_ba71_5c21513add9f.slice/crio-431843528eb39a6cc2824d5ac38291234083c70084f6a6662d0322d73aba356c WatchSource:0}: Error finding container 431843528eb39a6cc2824d5ac38291234083c70084f6a6662d0322d73aba356c: Status 404 returned error can't find the container with id 431843528eb39a6cc2824d5ac38291234083c70084f6a6662d0322d73aba356c
Sep 10 23:56:20.819910 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.819879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:20.925102 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.925077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:20.964100 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.964070    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"]
Sep 10 23:56:20.968831 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.968810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:20.971944 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.971923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6fa41e-dockercfg-jph28\""
Sep 10 23:56:20.972054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.971981    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6fa41e\""
Sep 10 23:56:20.986811 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.986164    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"]
Sep 10 23:56:20.994049 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.994022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r62bn\" (UniqueName: \"kubernetes.io/projected/3f07e4f5-d797-429c-bac9-91c4b9403aec-kube-api-access-r62bn\") pod \"payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:20.994183 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:20.994164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f07e4f5-d797-429c-bac9-91c4b9403aec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:21.095027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.094931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f07e4f5-d797-429c-bac9-91c4b9403aec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:21.095027 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.095004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r62bn\" (UniqueName: \"kubernetes.io/projected/3f07e4f5-d797-429c-bac9-91c4b9403aec-kube-api-access-r62bn\") pod \"payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:21.095663 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.095640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f07e4f5-d797-429c-bac9-91c4b9403aec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:21.103003 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.102981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r62bn\" (UniqueName: \"kubernetes.io/projected/3f07e4f5-d797-429c-bac9-91c4b9403aec-kube-api-access-r62bn\") pod \"payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:21.188101 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.188065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" event={"ID":"5c2c01a0-9311-4e14-8f82-09ea34e84b27","Type":"ContainerStarted","Data":"0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2"}
Sep 10 23:56:21.188242 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.188132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" event={"ID":"5c2c01a0-9311-4e14-8f82-09ea34e84b27","Type":"ContainerStarted","Data":"b951f174ba7efdb5c3d07af35cc9dfdf24649e89145bcd17aa30c66605a61007"}
Sep 10 23:56:21.188242 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.188162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:21.189807 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.189783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675" event={"ID":"05595a49-13ac-4995-ba71-5c21513add9f","Type":"ContainerStarted","Data":"0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0"}
Sep 10 23:56:21.189918 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.189847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675" event={"ID":"05595a49-13ac-4995-ba71-5c21513add9f","Type":"ContainerStarted","Data":"431843528eb39a6cc2824d5ac38291234083c70084f6a6662d0322d73aba356c"}
Sep 10 23:56:21.189918 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.189870    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:21.206632 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.206590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" podStartSLOduration=1.832427982 podStartE2EDuration="2.206575534s" totalImagesPullingTime="374.147552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:21.205199224 +0000 UTC m=+1223.161431026" watchObservedRunningTime="2026-09-10 23:56:21.206575534 +0000 UTC m=+1223.162807336"
Sep 10 23:56:21.207937 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.207909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:21.224268 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.224227    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675" podStartSLOduration=2.224215385 podStartE2EDuration="2.224215385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:21.223483448 +0000 UTC m=+1223.179715276" watchObservedRunningTime="2026-09-10 23:56:21.224215385 +0000 UTC m=+1223.180447188"
Sep 10 23:56:21.282989 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.282965    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:21.553847 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.553819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"]
Sep 10 23:56:21.556074 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:21.556044    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f07e4f5_d797_429c_bac9_91c4b9403aec.slice/crio-25a184e11a9150933d0f88aaebd6cb2c8e14136f8fef66dd97f91ce20dd1d6fc WatchSource:0}: Error finding container 25a184e11a9150933d0f88aaebd6cb2c8e14136f8fef66dd97f91ce20dd1d6fc: Status 404 returned error can't find the container with id 25a184e11a9150933d0f88aaebd6cb2c8e14136f8fef66dd97f91ce20dd1d6fc
Sep 10 23:56:21.881853 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:21.881770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:22.200604 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:22.200515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b" event={"ID":"3f07e4f5-d797-429c-bac9-91c4b9403aec","Type":"ContainerStarted","Data":"4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6"}
Sep 10 23:56:22.200604 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:22.200566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b" event={"ID":"3f07e4f5-d797-429c-bac9-91c4b9403aec","Type":"ContainerStarted","Data":"25a184e11a9150933d0f88aaebd6cb2c8e14136f8fef66dd97f91ce20dd1d6fc"}
Sep 10 23:56:22.218120 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:22.218062    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b" podStartSLOduration=2.218042293 podStartE2EDuration="2.218042293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:22.215121726 +0000 UTC m=+1224.171353529" watchObservedRunningTime="2026-09-10 23:56:22.218042293 +0000 UTC m=+1224.174274100"
Sep 10 23:56:22.541492 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:22.541449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:23.205129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:23.205095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:23.207178 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:23.207149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:25.445179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:25.445148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:56:27.206315 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:27.206284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:56:27.763403 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:27.763367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:27.819510 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:27.819466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:32.202777 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:32.202748    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:56:34.212496 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:34.212458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 10 23:56:36.292879 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.292845    2578 generic.go:356] "Generic (PLEG): container finished" podID="a3842711-b605-4843-9039-bb0440effb50" containerID="414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f" exitCode=0
Sep 10 23:56:36.293419 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.292931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" event={"ID":"a3842711-b605-4843-9039-bb0440effb50","Type":"ContainerDied","Data":"414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f"}
Sep 10 23:56:36.293484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.293432    2578 scope.go:117] "RemoveContainer" containerID="414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f"
Sep 10 23:56:36.294576 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.294550    2578 generic.go:356] "Generic (PLEG): container finished" podID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerID="4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80" exitCode=0
Sep 10 23:56:36.294679 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.294594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" event={"ID":"30ede721-c87a-419c-89b0-3ef6dbbab219","Type":"ContainerDied","Data":"4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80"}
Sep 10 23:56:36.295055 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.295040    2578 scope.go:117] "RemoveContainer" containerID="4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80"
Sep 10 23:56:36.295963 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.295937    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:36.296024 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.296002    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:36.296059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.296035    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:36.297066 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.297040    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:36.297129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.297098    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:36.297167 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:36.297128    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:56:37.301861 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:37.301828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" event={"ID":"30ede721-c87a-419c-89b0-3ef6dbbab219","Type":"ContainerStarted","Data":"56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569"}
Sep 10 23:56:37.303770 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:37.303747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" event={"ID":"a3842711-b605-4843-9039-bb0440effb50","Type":"ContainerStarted","Data":"c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3"}
Sep 10 23:56:37.881344 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:37.881180    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:37.881344 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:37.881339    2578 patch_prober.go:28] interesting pod/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 23:56:37.881639 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:37.881410    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 23:56:38.055179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:38.055132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:38.055604 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:38.055573    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 23:56:38.055736 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:38.055631    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 23:56:38.881299 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:38.880841    2578 patch_prober.go:28] interesting pod/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 23:56:38.881299 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:38.880906    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 23:56:39.055255 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:39.055202    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 23:56:39.055548 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:39.055304    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 23:56:39.880873 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:39.880820    2578 patch_prober.go:28] interesting pod/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 23:56:39.881061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:39.880912    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 23:56:40.055640 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:40.055591    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 23:56:40.056153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:40.055664    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 23:56:40.205207 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:40.205103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:40.880655 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:40.880608    2578 patch_prober.go:28] interesting pod/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 23:56:40.880870 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:40.880680    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 23:56:41.055769 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:41.055710    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 23:56:41.056374 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:41.056340    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 23:56:41.454369 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:41.454327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:41.880726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:41.880675    2578 patch_prober.go:28] interesting pod/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 23:56:41.880943 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:41.880747    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 23:56:41.998738 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:41.998705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:42.055929 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:42.055893    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 23:56:42.056538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:42.056505    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 23:56:42.278891 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:42.278849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:42.880619 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:42.880580    2578 patch_prober.go:28] interesting pod/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 10 23:56:42.880792 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:42.880637    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 10 23:56:42.957153 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:42.957114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:43.059674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:43.059628    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:43.060172 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:43.059890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:43.060663 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:43.060640    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 10 23:56:43.884308 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:43.884261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:43.884693 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:43.884672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:43.885359 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:43.885339    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:56:44.537331 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.537292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:44.661064 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.661037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:44.836333 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.836251    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"]
Sep 10 23:56:44.845732 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.845706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:44.847138 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.847110    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"]
Sep 10 23:56:44.849788 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.849763    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f8d9df10\"/\"openshift-service-ca.crt\""
Sep 10 23:56:44.850335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.849963    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f8d9df10\"/\"e2e-lifecycle-model-f8d9df10-kserve-self-signed-certs\""
Sep 10 23:56:44.850335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.850021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f8d9df10\"/\"kube-root-ca.crt\""
Sep 10 23:56:44.850335 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.849962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f8d9df10\"/\"default-dockercfg-w5qhk\""
Sep 10 23:56:44.853706 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.853660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:44.940379 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.940351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tls-certs\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:44.940516 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.940387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-model-cache\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:44.940516 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.940406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tmp-dir\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:44.940516 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.940420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-dshm\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:44.940633 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.940520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87kt\" (UniqueName: \"kubernetes.io/projected/c9578fd6-abb6-4141-91ec-cb99f6d41a95-kube-api-access-d87kt\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:44.940633 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:44.940546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-home\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.041627 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.041591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-model-cache\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.041627 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.041627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tmp-dir\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.041869 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.041643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-dshm\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.041869 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.041696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87kt\" (UniqueName: \"kubernetes.io/projected/c9578fd6-abb6-4141-91ec-cb99f6d41a95-kube-api-access-d87kt\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.041869 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.041717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-home\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.041869 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.041833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tls-certs\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.042120 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.042083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-model-cache\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.042251 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.042162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tmp-dir\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.042338 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.042265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-home\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.044019 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.043990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-dshm\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.044244 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.044226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tls-certs\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.052704 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.052681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87kt\" (UniqueName: \"kubernetes.io/projected/c9578fd6-abb6-4141-91ec-cb99f6d41a95-kube-api-access-d87kt\") pod \"e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") " pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.158342 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.158245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:45.429975 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.429950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"]
Sep 10 23:56:45.432019 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:45.431990    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9578fd6_abb6_4141_91ec_cb99f6d41a95.slice/crio-b138c6ddbb2c95f081effd66f4ee2fc7fe4c8b9a26acec542c1f7f61184a78ae WatchSource:0}: Error finding container b138c6ddbb2c95f081effd66f4ee2fc7fe4c8b9a26acec542c1f7f61184a78ae: Status 404 returned error can't find the container with id b138c6ddbb2c95f081effd66f4ee2fc7fe4c8b9a26acec542c1f7f61184a78ae
Sep 10 23:56:45.588980 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:45.588943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:46.227227 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:46.227192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:46.363462 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:46.363429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997" event={"ID":"c9578fd6-abb6-4141-91ec-cb99f6d41a95","Type":"ContainerStarted","Data":"76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0"}
Sep 10 23:56:46.363617 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:46.363468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997" event={"ID":"c9578fd6-abb6-4141-91ec-cb99f6d41a95","Type":"ContainerStarted","Data":"b138c6ddbb2c95f081effd66f4ee2fc7fe4c8b9a26acec542c1f7f61184a78ae"}
Sep 10 23:56:46.381978 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:46.381927    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997" podStartSLOduration=2.381913604 podStartE2EDuration="2.381913604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:46.379874978 +0000 UTC m=+1248.336106781" watchObservedRunningTime="2026-09-10 23:56:46.381913604 +0000 UTC m=+1248.338145407"
Sep 10 23:56:47.097836 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:47.097805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:47.203221 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:47.203189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:47.596172 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:47.596135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:47.950150 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:47.950071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:48.283330 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:48.283256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:48.385113 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:48.385083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:48.698385 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:48.698299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:56:50.662661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:50.662630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:50.888903 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:50.888863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:50.989635 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:50.989522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:51.566013 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:51.565984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:52.373023 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:52.372987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:53.062685 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.062653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:53.737932 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.737897    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"]
Sep 10 23:56:53.743297 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.743258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.745725 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.745695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-37c598-serving-cert\""
Sep 10 23:56:53.752657 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.752632    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"]
Sep 10 23:56:53.766745 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.766719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:53.818446 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.818420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkvl\" (UniqueName: \"kubernetes.io/projected/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-kube-api-access-cmkvl\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.818551 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.818492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.818551 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.818546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.818632 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.818578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.868824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.868796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:53.919200 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.919176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.919341 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.919241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.919341 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.919262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.919341 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.919313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkvl\" (UniqueName: \"kubernetes.io/projected/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-kube-api-access-cmkvl\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.920033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.920014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.921538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.921517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.921614 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.921578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.926572 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.926546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkvl\" (UniqueName: \"kubernetes.io/projected/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-kube-api-access-cmkvl\") pod \"maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:53.970124 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:53.970093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:54.057444 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:54.057408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:54.216425 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:54.216397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:54.312541 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:54.312462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:54.337905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:54.337873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"]
Sep 10 23:56:54.339502 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:56:54.339474    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577e0dfc_7d72_47e5_94f9_c8db6c66ef39.slice/crio-0ae6a78a3756ed2e08f73bae7a205df0477ad75d0a251dff464f5dc588c5a9ad WatchSource:0}: Error finding container 0ae6a78a3756ed2e08f73bae7a205df0477ad75d0a251dff464f5dc588c5a9ad: Status 404 returned error can't find the container with id 0ae6a78a3756ed2e08f73bae7a205df0477ad75d0a251dff464f5dc588c5a9ad
Sep 10 23:56:54.405484 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:54.405450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l" event={"ID":"577e0dfc-7d72-47e5-94f9-c8db6c66ef39","Type":"ContainerStarted","Data":"0ae6a78a3756ed2e08f73bae7a205df0477ad75d0a251dff464f5dc588c5a9ad"}
Sep 10 23:56:55.159097 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:55.158990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:55.159097 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:55.159049    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:55.171714 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:55.171689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:55.414168 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:55.414065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l" event={"ID":"577e0dfc-7d72-47e5-94f9-c8db6c66ef39","Type":"ContainerStarted","Data":"936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6"}
Sep 10 23:56:55.414367 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:55.414348    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:56:55.425267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:55.425244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:56:55.432541 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:55.432488    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l" podStartSLOduration=2.007870786 podStartE2EDuration="2.432471272s" totalImagesPullingTime="424.600486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:55.431356204 +0000 UTC m=+1257.387588008" watchObservedRunningTime="2026-09-10 23:56:55.432471272 +0000 UTC m=+1257.388703078"
Sep 10 23:56:59.409392 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:59.409350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:56:59.986955 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:56:59.986915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:01.424370 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:01.424336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 10 23:57:11.584450 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:11.584412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:57:14.612689 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:14.612652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:15.878863 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:15.878831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:16.253041 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:16.252918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:17.088940 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:17.088909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:17.842780 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:17.842745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:18.656802 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:18.656767    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:19.362592 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:19.362555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:19.990237 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:19.990205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:20.782612 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:20.782572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:21.640782 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.640744    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"]
Sep 10 23:57:21.645967 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.645947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.650590 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.650494    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"openshift-service-ca.crt\""
Sep 10 23:57:21.650590 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.650583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"auth-test-model-6fa41e-kserve-self-signed-certs\""
Sep 10 23:57:21.650803 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.650646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"default-dockercfg-rnn8j\""
Sep 10 23:57:21.650865 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.650787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"kube-root-ca.crt\""
Sep 10 23:57:21.653329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.653304    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"]
Sep 10 23:57:21.730610 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.730579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:21.786821 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.786790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b7e23c-f38b-4281-a3d0-81d0992541be-tls-certs\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.787006 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.786826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6rrj\" (UniqueName: \"kubernetes.io/projected/21b7e23c-f38b-4281-a3d0-81d0992541be-kube-api-access-v6rrj\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.787006 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.786848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-home\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.787006 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.786891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-model-cache\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.787006 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.786975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-tmp-dir\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.787147 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.787043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-dshm\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.887631 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.887594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b7e23c-f38b-4281-a3d0-81d0992541be-tls-certs\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.887631 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.887632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rrj\" (UniqueName: \"kubernetes.io/projected/21b7e23c-f38b-4281-a3d0-81d0992541be-kube-api-access-v6rrj\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.887900 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.887651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-home\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.887900 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.887672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-model-cache\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.887900 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.887708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-tmp-dir\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.887900 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.887764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-dshm\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.888126 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.888085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-home\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.888126 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.888113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-model-cache\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.888205 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.888169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-tmp-dir\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.890016 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.889996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-dshm\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.890202 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.890181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b7e23c-f38b-4281-a3d0-81d0992541be-tls-certs\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.896100 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.896039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rrj\" (UniqueName: \"kubernetes.io/projected/21b7e23c-f38b-4281-a3d0-81d0992541be-kube-api-access-v6rrj\") pod \"auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:21.957497 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:21.957457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:22.229162 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:22.229134    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"]
Sep 10 23:57:22.230661 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:57:22.230634    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b7e23c_f38b_4281_a3d0_81d0992541be.slice/crio-0cd997d8657845b7e9d307653356bd73647cb19a068c3f8789b1fe6781594c54 WatchSource:0}: Error finding container 0cd997d8657845b7e9d307653356bd73647cb19a068c3f8789b1fe6781594c54: Status 404 returned error can't find the container with id 0cd997d8657845b7e9d307653356bd73647cb19a068c3f8789b1fe6781594c54
Sep 10 23:57:22.559611 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:22.559566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7" event={"ID":"21b7e23c-f38b-4281-a3d0-81d0992541be","Type":"ContainerStarted","Data":"df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e"}
Sep 10 23:57:22.559797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:22.559620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7" event={"ID":"21b7e23c-f38b-4281-a3d0-81d0992541be","Type":"ContainerStarted","Data":"0cd997d8657845b7e9d307653356bd73647cb19a068c3f8789b1fe6781594c54"}
Sep 10 23:57:22.576453 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:22.576399    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7" podStartSLOduration=1.576383763 podStartE2EDuration="1.576383763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:57:22.574620662 +0000 UTC m=+1284.530852468" watchObservedRunningTime="2026-09-10 23:57:22.576383763 +0000 UTC m=+1284.532615565"
Sep 10 23:57:26.422939 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:26.422902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:27.503323 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:27.503284    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:57:31.957630 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:31.957592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:31.958096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:31.957646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:31.975542 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:31.975506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:31.979806 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:31.979776    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"]
Sep 10 23:57:31.980071 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:31.980022    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" podUID="5c2c01a0-9311-4e14-8f82-09ea34e84b27" containerName="maas-api" containerID="cri-o://0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2" gracePeriod=30
Sep 10 23:57:32.007058 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.007012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:32.041438 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.041397    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"]
Sep 10 23:57:32.041677 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.041651    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675" podUID="05595a49-13ac-4995-ba71-5c21513add9f" containerName="payload-pre-processing" containerID="cri-o://0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0" gracePeriod=30
Sep 10 23:57:32.106818 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.106787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:32.201549 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.201506    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675" podUID="05595a49-13ac-4995-ba71-5c21513add9f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.59:9004: connect: connection refused"
Sep 10 23:57:32.201735 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.201558    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" podUID="5c2c01a0-9311-4e14-8f82-09ea34e84b27" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": dial tcp 10.132.0.58:8443: connect: connection refused"
Sep 10 23:57:32.202781 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.202702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:32.264858 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.264602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:32.310849 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.310815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:57:32.335652 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.335625    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:57:32.489632 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.489530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05595a49-13ac-4995-ba71-5c21513add9f-plugins-config-volume\") pod \"05595a49-13ac-4995-ba71-5c21513add9f\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") "
Sep 10 23:57:32.489632 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.489593    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls\") pod \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") "
Sep 10 23:57:32.489894 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.489682    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92tb7\" (UniqueName: \"kubernetes.io/projected/5c2c01a0-9311-4e14-8f82-09ea34e84b27-kube-api-access-92tb7\") pod \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") "
Sep 10 23:57:32.489894 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.489746    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2c01a0-9311-4e14-8f82-09ea34e84b27-openshift-service-ca\") pod \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") "
Sep 10 23:57:32.489894 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.489781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-metrics-certs\") pod \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\" (UID: \"5c2c01a0-9311-4e14-8f82-09ea34e84b27\") "
Sep 10 23:57:32.489894 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.489837    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl82k\" (UniqueName: \"kubernetes.io/projected/05595a49-13ac-4995-ba71-5c21513add9f-kube-api-access-bl82k\") pod \"05595a49-13ac-4995-ba71-5c21513add9f\" (UID: \"05595a49-13ac-4995-ba71-5c21513add9f\") "
Sep 10 23:57:32.490186 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.490006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05595a49-13ac-4995-ba71-5c21513add9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05595a49-13ac-4995-ba71-5c21513add9f" (UID: "05595a49-13ac-4995-ba71-5c21513add9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:57:32.490186 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.490164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2c01a0-9311-4e14-8f82-09ea34e84b27-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c2c01a0-9311-4e14-8f82-09ea34e84b27" (UID: "5c2c01a0-9311-4e14-8f82-09ea34e84b27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:57:32.490327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.490173    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05595a49-13ac-4995-ba71-5c21513add9f-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:32.491844 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.491815    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c2c01a0-9311-4e14-8f82-09ea34e84b27" (UID: "5c2c01a0-9311-4e14-8f82-09ea34e84b27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:57:32.491946 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.491866    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2c01a0-9311-4e14-8f82-09ea34e84b27-kube-api-access-92tb7" (OuterVolumeSpecName: "kube-api-access-92tb7") pod "5c2c01a0-9311-4e14-8f82-09ea34e84b27" (UID: "5c2c01a0-9311-4e14-8f82-09ea34e84b27"). InnerVolumeSpecName "kube-api-access-92tb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:57:32.491946 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.491900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c2c01a0-9311-4e14-8f82-09ea34e84b27" (UID: "5c2c01a0-9311-4e14-8f82-09ea34e84b27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:57:32.492019 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.491950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05595a49-13ac-4995-ba71-5c21513add9f-kube-api-access-bl82k" (OuterVolumeSpecName: "kube-api-access-bl82k") pod "05595a49-13ac-4995-ba71-5c21513add9f" (UID: "05595a49-13ac-4995-ba71-5c21513add9f"). InnerVolumeSpecName "kube-api-access-bl82k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:57:32.585914 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.585861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:32.591033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.590997    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-metrics-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:32.591033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.591033    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl82k\" (UniqueName: \"kubernetes.io/projected/05595a49-13ac-4995-ba71-5c21513add9f-kube-api-access-bl82k\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:32.591198 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.591048    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2c01a0-9311-4e14-8f82-09ea34e84b27-maas-api-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:32.591198 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.591060    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92tb7\" (UniqueName: \"kubernetes.io/projected/5c2c01a0-9311-4e14-8f82-09ea34e84b27-kube-api-access-92tb7\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:32.591198 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.591071    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2c01a0-9311-4e14-8f82-09ea34e84b27-openshift-service-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:32.610426 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.610396    2578 generic.go:356] "Generic (PLEG): container finished" podID="05595a49-13ac-4995-ba71-5c21513add9f" containerID="0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0" exitCode=0
Sep 10 23:57:32.610570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.610456    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"
Sep 10 23:57:32.610570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.610487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675" event={"ID":"05595a49-13ac-4995-ba71-5c21513add9f","Type":"ContainerDied","Data":"0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0"}
Sep 10 23:57:32.610570 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.610549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675" event={"ID":"05595a49-13ac-4995-ba71-5c21513add9f","Type":"ContainerDied","Data":"431843528eb39a6cc2824d5ac38291234083c70084f6a6662d0322d73aba356c"}
Sep 10 23:57:32.610743 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.610578    2578 scope.go:117] "RemoveContainer" containerID="0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0"
Sep 10 23:57:32.612079 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.612045    2578 generic.go:356] "Generic (PLEG): container finished" podID="5c2c01a0-9311-4e14-8f82-09ea34e84b27" containerID="0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2" exitCode=0
Sep 10 23:57:32.612175 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.612104    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"
Sep 10 23:57:32.612175 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.612114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" event={"ID":"5c2c01a0-9311-4e14-8f82-09ea34e84b27","Type":"ContainerDied","Data":"0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2"}
Sep 10 23:57:32.612175 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.612167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz" event={"ID":"5c2c01a0-9311-4e14-8f82-09ea34e84b27","Type":"ContainerDied","Data":"b951f174ba7efdb5c3d07af35cc9dfdf24649e89145bcd17aa30c66605a61007"}
Sep 10 23:57:32.620990 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.620973    2578 scope.go:117] "RemoveContainer" containerID="0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0"
Sep 10 23:57:32.621317 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:57:32.621263    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0\": container with ID starting with 0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0 not found: ID does not exist" containerID="0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0"
Sep 10 23:57:32.621394 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.621327    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0"} err="failed to get container status \"0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0\": rpc error: code = NotFound desc = could not find container \"0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0\": container with ID starting with 0d1309d0a0edb40c208fa796aa5c9cbab6247b439d448991e27b30af425a9fb0 not found: ID does not exist"
Sep 10 23:57:32.621394 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.621352    2578 scope.go:117] "RemoveContainer" containerID="0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2"
Sep 10 23:57:32.626719 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.626696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 10 23:57:32.629678 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.629658    2578 scope.go:117] "RemoveContainer" containerID="0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2"
Sep 10 23:57:32.630035 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:57:32.630009    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2\": container with ID starting with 0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2 not found: ID does not exist" containerID="0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2"
Sep 10 23:57:32.630035 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.630044    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2"} err="failed to get container status \"0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2\": rpc error: code = NotFound desc = could not find container \"0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2\": container with ID starting with 0bc2eae08afaf40b81bcff6907ab3361269147edcc828ed8db6d4ea83ccd16c2 not found: ID does not exist"
Sep 10 23:57:32.632504 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.632482    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"]
Sep 10 23:57:32.641420 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.641396    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8d9df10-5f87dd587d-5b675"]
Sep 10 23:57:32.654183 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.654156    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"]
Sep 10 23:57:32.658590 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:32.658559    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8d9df10-697569f8f5-lwshz"]
Sep 10 23:57:33.452978 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:33.452920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:34.100513 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:34.100480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:34.534483 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:34.534443    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05595a49-13ac-4995-ba71-5c21513add9f" path="/var/lib/kubelet/pods/05595a49-13ac-4995-ba71-5c21513add9f/volumes"
Sep 10 23:57:34.535033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:34.535010    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c2c01a0-9311-4e14-8f82-09ea34e84b27" path="/var/lib/kubelet/pods/5c2c01a0-9311-4e14-8f82-09ea34e84b27/volumes"
Sep 10 23:57:34.950540 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:34.950453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:35.276883 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:35.276850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:35.692820 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:35.692732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:40.257007 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:40.256973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"]
Sep 10 23:57:40.304599 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:40.304566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:44.384915 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:44.384877    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:57:47.238784 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.238750    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"]
Sep 10 23:57:47.239179 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.239164    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2c01a0-9311-4e14-8f82-09ea34e84b27" containerName="maas-api"
Sep 10 23:57:47.239224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.239181    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2c01a0-9311-4e14-8f82-09ea34e84b27" containerName="maas-api"
Sep 10 23:57:47.239224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.239192    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05595a49-13ac-4995-ba71-5c21513add9f" containerName="payload-pre-processing"
Sep 10 23:57:47.239224 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.239201    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="05595a49-13ac-4995-ba71-5c21513add9f" containerName="payload-pre-processing"
Sep 10 23:57:47.239339 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.239325    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2c01a0-9311-4e14-8f82-09ea34e84b27" containerName="maas-api"
Sep 10 23:57:47.239339 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.239338    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="05595a49-13ac-4995-ba71-5c21513add9f" containerName="payload-pre-processing"
Sep 10 23:57:47.243958 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.243936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.247177 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.247150    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"auth-test-model-37c598-kserve-self-signed-certs\""
Sep 10 23:57:47.248204 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.248181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"default-dockercfg-rxxx8\""
Sep 10 23:57:47.248389 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.248190    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"kube-root-ca.crt\""
Sep 10 23:57:47.248539 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.248251    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"openshift-service-ca.crt\""
Sep 10 23:57:47.253958 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.253928    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"]
Sep 10 23:57:47.257209 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.257183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:47.328218 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.328187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tls-certs\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.328404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.328232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-home\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.328404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.328256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-model-cache\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.328404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.328348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tmp-dir\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.328404 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.328385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-dshm\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.328575 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.328441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb866\" (UniqueName: \"kubernetes.io/projected/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-kube-api-access-pb866\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.429452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.429409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tls-certs\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.429618 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.429485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-home\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.429618 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.429519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-model-cache\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.429618 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.429573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tmp-dir\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.429618 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.429609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-dshm\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.429837 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.429694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb866\" (UniqueName: \"kubernetes.io/projected/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-kube-api-access-pb866\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.430009 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.429981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-home\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.430104 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.430011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tmp-dir\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.430104 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.430047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-model-cache\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.431901 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.431876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-dshm\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.432033 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.432016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tls-certs\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.438269 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.438246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb866\" (UniqueName: \"kubernetes.io/projected/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-kube-api-access-pb866\") pod \"auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") " pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.556872 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.556833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:47.829318 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:47.829265    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"]
Sep 10 23:57:47.831834 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:57:47.831791    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c713a79_2ae0_4a6c_b727_ebc0f20681dc.slice/crio-ccbc3665acd6b857fe4f4240c939600e2a2dea45982f2872553d32cb9e60c46f WatchSource:0}: Error finding container ccbc3665acd6b857fe4f4240c939600e2a2dea45982f2872553d32cb9e60c46f: Status 404 returned error can't find the container with id ccbc3665acd6b857fe4f4240c939600e2a2dea45982f2872553d32cb9e60c46f
Sep 10 23:57:48.698758 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:48.698716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8" event={"ID":"2c713a79-2ae0-4a6c-b727-ebc0f20681dc","Type":"ContainerStarted","Data":"53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9"}
Sep 10 23:57:48.699154 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:48.698764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8" event={"ID":"2c713a79-2ae0-4a6c-b727-ebc0f20681dc","Type":"ContainerStarted","Data":"ccbc3665acd6b857fe4f4240c939600e2a2dea45982f2872553d32cb9e60c46f"}
Sep 10 23:57:48.718880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:48.718833    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8" podStartSLOduration=1.718815086 podStartE2EDuration="1.718815086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:57:48.717479152 +0000 UTC m=+1310.673710956" watchObservedRunningTime="2026-09-10 23:57:48.718815086 +0000 UTC m=+1310.675046888"
Sep 10 23:57:55.278575 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.278535    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997" podUID="c9578fd6-abb6-4141-91ec-cb99f6d41a95" containerName="main" containerID="cri-o://76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0" gracePeriod=45
Sep 10 23:57:55.415170 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.415134    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997" podUID="c9578fd6-abb6-4141-91ec-cb99f6d41a95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 10 23:57:55.526741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.526713    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:57:55.611219 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tls-certs\") pod \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") "
Sep 10 23:57:55.611219 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tmp-dir\") pod \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") "
Sep 10 23:57:55.611219 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-dshm\") pod \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") "
Sep 10 23:57:55.611565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-home\") pod \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") "
Sep 10 23:57:55.611565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d87kt\" (UniqueName: \"kubernetes.io/projected/c9578fd6-abb6-4141-91ec-cb99f6d41a95-kube-api-access-d87kt\") pod \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") "
Sep 10 23:57:55.611565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-model-cache\") pod \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\" (UID: \"c9578fd6-abb6-4141-91ec-cb99f6d41a95\") "
Sep 10 23:57:55.611565 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9578fd6-abb6-4141-91ec-cb99f6d41a95" (UID: "c9578fd6-abb6-4141-91ec-cb99f6d41a95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:55.611741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611569    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-home" (OuterVolumeSpecName: "home") pod "c9578fd6-abb6-4141-91ec-cb99f6d41a95" (UID: "c9578fd6-abb6-4141-91ec-cb99f6d41a95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:55.611741 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9578fd6-abb6-4141-91ec-cb99f6d41a95" (UID: "c9578fd6-abb6-4141-91ec-cb99f6d41a95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:55.611818 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611795    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:55.611856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611816    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:55.611856 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.611833    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:55.613434 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.613413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-dshm" (OuterVolumeSpecName: "dshm") pod "c9578fd6-abb6-4141-91ec-cb99f6d41a95" (UID: "c9578fd6-abb6-4141-91ec-cb99f6d41a95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:55.613536 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.613517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9578fd6-abb6-4141-91ec-cb99f6d41a95" (UID: "c9578fd6-abb6-4141-91ec-cb99f6d41a95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:57:55.613586 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.613547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9578fd6-abb6-4141-91ec-cb99f6d41a95-kube-api-access-d87kt" (OuterVolumeSpecName: "kube-api-access-d87kt") pod "c9578fd6-abb6-4141-91ec-cb99f6d41a95" (UID: "c9578fd6-abb6-4141-91ec-cb99f6d41a95"). InnerVolumeSpecName "kube-api-access-d87kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:57:55.713338 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.713309    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9578fd6-abb6-4141-91ec-cb99f6d41a95-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:55.713338 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.713337    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9578fd6-abb6-4141-91ec-cb99f6d41a95-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:55.713490 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.713348    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d87kt\" (UniqueName: \"kubernetes.io/projected/c9578fd6-abb6-4141-91ec-cb99f6d41a95-kube-api-access-d87kt\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:57:55.734173 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.734149    2578 generic.go:356] "Generic (PLEG): container finished" podID="c9578fd6-abb6-4141-91ec-cb99f6d41a95" containerID="76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0" exitCode=0
Sep 10 23:57:55.734327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.734204    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"
Sep 10 23:57:55.734327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.734214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997" event={"ID":"c9578fd6-abb6-4141-91ec-cb99f6d41a95","Type":"ContainerDied","Data":"76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0"}
Sep 10 23:57:55.734327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.734250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997" event={"ID":"c9578fd6-abb6-4141-91ec-cb99f6d41a95","Type":"ContainerDied","Data":"b138c6ddbb2c95f081effd66f4ee2fc7fe4c8b9a26acec542c1f7f61184a78ae"}
Sep 10 23:57:55.734327 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.734267    2578 scope.go:117] "RemoveContainer" containerID="76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0"
Sep 10 23:57:55.743329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.743311    2578 scope.go:117] "RemoveContainer" containerID="76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0"
Sep 10 23:57:55.743613 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:57:55.743593    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0\": container with ID starting with 76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0 not found: ID does not exist" containerID="76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0"
Sep 10 23:57:55.743727 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.743629    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0"} err="failed to get container status \"76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0\": rpc error: code = NotFound desc = could not find container \"76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0\": container with ID starting with 76fd6ddb4b82f50dae94738f2545b1df7f4eb612471eb055bdd51c8af09bc4d0 not found: ID does not exist"
Sep 10 23:57:55.758413 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.758392    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"]
Sep 10 23:57:55.763190 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:55.763171    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f8d9df10/e2e-lifecycle-model-f8d9df10-kserve-796fb446ff-gk997"]
Sep 10 23:57:56.533803 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:56.533771    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9578fd6-abb6-4141-91ec-cb99f6d41a95" path="/var/lib/kubelet/pods/c9578fd6-abb6-4141-91ec-cb99f6d41a95/volumes"
Sep 10 23:57:57.557214 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:57.557181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:57.557612 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:57.557225    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:57.569800 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:57.569774    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:57.759004 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:57.758978    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 10 23:57:59.481996 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:57:59.481959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:58:12.885731 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:12.885695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:58:17.349684 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:17.349648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:17.707007 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:17.706914    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:23.137269 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:23.137188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:28.688543 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:28.688508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:28.788600 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:28.788568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:28.919963 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:28.919929    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"]
Sep 10 23:58:28.920240 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:28.920212    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy" containerID="cri-o://c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3" gracePeriod=30
Sep 10 23:58:28.940863 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:28.940792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:30.088202 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:30.088166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:58:30.128182 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:30.128148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:30.226244 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:30.226211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:30.322609 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:30.322581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:30.946211 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:30.946180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:31.040796 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:31.040746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:31.149176 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:31.149144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:32.341236 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:32.341191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:34.160909 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.160882    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:58:34.269106 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269074    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3842711-b605-4843-9039-bb0440effb50-istiod-ca-cert\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3842711-b605-4843-9039-bb0440effb50-istio-podinfo\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-istio-token\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-socket\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269267 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-certs\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269528 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxxm4\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-kube-api-access-wxxm4\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269528 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-envoy\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269528 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269400    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-credential-socket\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269528 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-data\") pod \"a3842711-b605-4843-9039-bb0440effb50\" (UID: \"a3842711-b605-4843-9039-bb0440effb50\") "
Sep 10 23:58:34.269745 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:34.269745 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:34.269745 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3842711-b605-4843-9039-bb0440effb50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:58:34.269905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:34.269905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:34.269905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269854    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3842711-b605-4843-9039-bb0440effb50-istiod-ca-cert\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.269905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269875    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.269905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.269886    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-workload-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.271506 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.271475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:58:34.271613 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.271531    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-kube-api-access-wxxm4" (OuterVolumeSpecName: "kube-api-access-wxxm4") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "kube-api-access-wxxm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:58:34.271755 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.271723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3842711-b605-4843-9039-bb0440effb50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:58:34.271755 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.271731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3842711-b605-4843-9039-bb0440effb50" (UID: "a3842711-b605-4843-9039-bb0440effb50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:34.370538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.370499    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3842711-b605-4843-9039-bb0440effb50-istio-podinfo\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.370538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.370530    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-istio-token\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.370538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.370538    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxxm4\" (UniqueName: \"kubernetes.io/projected/a3842711-b605-4843-9039-bb0440effb50-kube-api-access-wxxm4\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.370538 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.370547    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-envoy\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.370797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.370556    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-credential-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.370797 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.370564    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3842711-b605-4843-9039-bb0440effb50-istio-data\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 10 23:58:34.942223 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.942129    2578 generic.go:356] "Generic (PLEG): container finished" podID="a3842711-b605-4843-9039-bb0440effb50" containerID="c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3" exitCode=0
Sep 10 23:58:34.942223 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.942176    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"
Sep 10 23:58:34.942223 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.942208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" event={"ID":"a3842711-b605-4843-9039-bb0440effb50","Type":"ContainerDied","Data":"c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3"}
Sep 10 23:58:34.942502 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.942261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9" event={"ID":"a3842711-b605-4843-9039-bb0440effb50","Type":"ContainerDied","Data":"3d2ad331c73b9f73e732587882b6509cb8193cd9eeb89fdd5018be0a25a881af"}
Sep 10 23:58:34.942502 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.942311    2578 scope.go:117] "RemoveContainer" containerID="c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3"
Sep 10 23:58:34.951220 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.951203    2578 scope.go:117] "RemoveContainer" containerID="414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f"
Sep 10 23:58:34.959461 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.959434    2578 scope.go:117] "RemoveContainer" containerID="c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3"
Sep 10 23:58:34.959898 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:58:34.959871    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3\": container with ID starting with c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3 not found: ID does not exist" containerID="c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3"
Sep 10 23:58:34.960008 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.959909    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3"} err="failed to get container status \"c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3\": rpc error: code = NotFound desc = could not find container \"c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3\": container with ID starting with c151fde45eb9b82aeedb0fd761a90408d2343f3f1d02c470ed5b3e20f9e36dc3 not found: ID does not exist"
Sep 10 23:58:34.960008 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.959933    2578 scope.go:117] "RemoveContainer" containerID="414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f"
Sep 10 23:58:34.960645 ip-10-0-128-88 kubenswrapper[2578]: E0910 23:58:34.960594    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f\": container with ID starting with 414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f not found: ID does not exist" containerID="414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f"
Sep 10 23:58:34.960738 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.960654    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f"} err="failed to get container status \"414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f\": rpc error: code = NotFound desc = could not find container \"414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f\": container with ID starting with 414bc8bb5243b2c231e9ef752946174ca73da52bc8f18d00e4289a476813303f not found: ID does not exist"
Sep 10 23:58:34.962831 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.962809    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"]
Sep 10 23:58:34.968430 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:34.968409    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f8d9df10-openshift-default-86559cc88d-szzj9"]
Sep 10 23:58:36.534215 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:36.534177    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3842711-b605-4843-9039-bb0440effb50" path="/var/lib/kubelet/pods/a3842711-b605-4843-9039-bb0440effb50/volumes"
Sep 10 23:58:37.751475 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:37.751441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:38.043164 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:38.043131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:38.100385 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:38.100356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:38.704240 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:38.704196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:41.892045 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:41.892012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:58:44.521285 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:44.521236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:44.682654 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:44.682624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:44.777889 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:44.777797    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:55.212602 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:55.212565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:55.411674 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:55.411634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:57.592329 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:58:57.592292    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:59:12.592833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:12.592801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:59:16.305845 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:16.305804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:22.045833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.045799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:22.111003 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.110964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:22.371880 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.371785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:22.648574 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.648489    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"]
Sep 10 23:59:22.648955 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.648938    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy"
Sep 10 23:59:22.649051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.648959    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy"
Sep 10 23:59:22.649051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.648972    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy"
Sep 10 23:59:22.649051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.648980    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy"
Sep 10 23:59:22.649051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.649017    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9578fd6-abb6-4141-91ec-cb99f6d41a95" containerName="main"
Sep 10 23:59:22.649051 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.649026    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9578fd6-abb6-4141-91ec-cb99f6d41a95" containerName="main"
Sep 10 23:59:22.649322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.649120    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy"
Sep 10 23:59:22.649322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.649131    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9578fd6-abb6-4141-91ec-cb99f6d41a95" containerName="main"
Sep 10 23:59:22.649322 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.649143    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3842711-b605-4843-9039-bb0440effb50" containerName="istio-proxy"
Sep 10 23:59:22.653715 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.653694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.656439 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.656416    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-afe559-openshift-default-dockercfg-dzxm9\""
Sep 10 23:59:22.663378 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.663105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"]
Sep 10 23:59:22.674721 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.674694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:22.710587 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxff9\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-kube-api-access-rxff9\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-data\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-socket\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710726 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-istio-token\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710850 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-envoy\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710850 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334c6b48-0d1d-414e-9117-4c401741c484-istiod-ca-cert\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710850 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334c6b48-0d1d-414e-9117-4c401741c484-istio-podinfo\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710850 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-certs\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.710997 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.710870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-credential-socket\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.811993 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.811959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-socket\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-istio-token\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-envoy\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334c6b48-0d1d-414e-9117-4c401741c484-istiod-ca-cert\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334c6b48-0d1d-414e-9117-4c401741c484-istio-podinfo\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-certs\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812472 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-credential-socket\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812472 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxff9\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-kube-api-access-rxff9\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-socket\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-data\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-certs\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812608 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-credential-socket\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-data\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.812824 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.812817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334c6b48-0d1d-414e-9117-4c401741c484-istiod-ca-cert\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.814377 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.814358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-envoy\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.814630 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.814610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334c6b48-0d1d-414e-9117-4c401741c484-istio-podinfo\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.819971 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.819945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-istio-token\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.820522 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.820504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxff9\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-kube-api-access-rxff9\") pod \"e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") " pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:22.880756 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.880721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:22.968269 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:22.968185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:23.242368 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:23.242338    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"]
Sep 10 23:59:23.244371 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:59:23.244338    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334c6b48_0d1d_414e_9117_4c401741c484.slice/crio-8899ba7a15d78ebb2133434f64d6ed3619f6271266d03ae4d88b86ecd76e4213 WatchSource:0}: Error finding container 8899ba7a15d78ebb2133434f64d6ed3619f6271266d03ae4d88b86ecd76e4213: Status 404 returned error can't find the container with id 8899ba7a15d78ebb2133434f64d6ed3619f6271266d03ae4d88b86ecd76e4213
Sep 10 23:59:23.246811 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:23.246776    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:59:23.246898 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:23.246841    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:59:23.246898 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:23.246874    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:59:24.183665 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:24.183630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" event={"ID":"334c6b48-0d1d-414e-9117-4c401741c484","Type":"ContainerStarted","Data":"17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80"}
Sep 10 23:59:24.183861 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:24.183690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" event={"ID":"334c6b48-0d1d-414e-9117-4c401741c484","Type":"ContainerStarted","Data":"8899ba7a15d78ebb2133434f64d6ed3619f6271266d03ae4d88b86ecd76e4213"}
Sep 10 23:59:24.206340 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:24.206263    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" podStartSLOduration=2.206243052 podStartE2EDuration="2.206243052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:24.203969038 +0000 UTC m=+1406.160200841" watchObservedRunningTime="2026-09-10 23:59:24.206243052 +0000 UTC m=+1406.162474859"
Sep 10 23:59:24.968491 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:24.968456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:24.974059 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:24.974032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:25.190129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:25.190089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:25.191193 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:25.191172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:25.487922 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:25.487883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:26.082302 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:26.082248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:26.176042 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:26.176008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:26.275846 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:26.275814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:27.170793 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:27.170752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:28.004737 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:28.004703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:28.096661 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:28.096625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:59:28.673984 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:28.673941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:29.489452 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:29.489417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:29.589397 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:29.589361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:29.801684 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:29.801652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:29.954721 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:29.954690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:30.055129 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.055038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:30.096728 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.096692    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"]
Sep 10 23:59:30.100238 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.100219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.102749 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.102725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-afe559-dockercfg-jh5pf\""
Sep 10 23:59:30.102845 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.102751    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-afe559\""
Sep 10 23:59:30.108188 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.108161    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"]
Sep 10 23:59:30.182733 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.182696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-afe559-784f548b77-tkrz4\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.182905 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.182764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwr6c\" (UniqueName: \"kubernetes.io/projected/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-kube-api-access-mwr6c\") pod \"payload-processing-e2e-api-a-afe559-784f548b77-tkrz4\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.284234 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.284192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwr6c\" (UniqueName: \"kubernetes.io/projected/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-kube-api-access-mwr6c\") pod \"payload-processing-e2e-api-a-afe559-784f548b77-tkrz4\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.284435 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.284333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-afe559-784f548b77-tkrz4\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.284974 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.284954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-afe559-784f548b77-tkrz4\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.292862 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.292840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwr6c\" (UniqueName: \"kubernetes.io/projected/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-kube-api-access-mwr6c\") pod \"payload-processing-e2e-api-a-afe559-784f548b77-tkrz4\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.412710 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.412619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:30.684307 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:30.684243    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"]
Sep 10 23:59:30.686508 ip-10-0-128-88 kubenswrapper[2578]: W0910 23:59:30.686478    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e1ced1_d335_4f57_89bf_9ce7ed1dd4ed.slice/crio-cc7810ed0d8f377c77c45632dba81d268eb599d9f7b51261744d567006e1eecc WatchSource:0}: Error finding container cc7810ed0d8f377c77c45632dba81d268eb599d9f7b51261744d567006e1eecc: Status 404 returned error can't find the container with id cc7810ed0d8f377c77c45632dba81d268eb599d9f7b51261744d567006e1eecc
Sep 10 23:59:31.221405 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:31.221366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4" event={"ID":"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed","Type":"ContainerStarted","Data":"918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb"}
Sep 10 23:59:31.221833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:31.221431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4" event={"ID":"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed","Type":"ContainerStarted","Data":"cc7810ed0d8f377c77c45632dba81d268eb599d9f7b51261744d567006e1eecc"}
Sep 10 23:59:31.221833 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:31.221487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:31.238849 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:31.238787    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4" podStartSLOduration=1.238765873 podStartE2EDuration="1.238765873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:31.23704804 +0000 UTC m=+1413.193279842" watchObservedRunningTime="2026-09-10 23:59:31.238765873 +0000 UTC m=+1413.194997677"
Sep 10 23:59:42.228823 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:42.228785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 10 23:59:45.991247 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:45.991210    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 10 23:59:46.957582 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:46.957540    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"]
Sep 10 23:59:46.963855 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:46.963821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:47.307604 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:47.307570    2578 generic.go:356] "Generic (PLEG): container finished" podID="334c6b48-0d1d-414e-9117-4c401741c484" containerID="17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80" exitCode=0
Sep 10 23:59:47.308009 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:47.307644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" event={"ID":"334c6b48-0d1d-414e-9117-4c401741c484","Type":"ContainerDied","Data":"17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80"}
Sep 10 23:59:47.308096 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:47.308080    2578 scope.go:117] "RemoveContainer" containerID="17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80"
Sep 10 23:59:47.309349 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:47.309315    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:59:47.309441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:47.309392    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:59:47.309441 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:47.309424    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:59:48.061400 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:48.061361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"]
Sep 10 23:59:48.081054 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:48.081017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:48.314131 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:48.314045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" event={"ID":"334c6b48-0d1d-414e-9117-4c401741c484","Type":"ContainerStarted","Data":"2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798"}
Sep 10 23:59:48.968775 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:48.968741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:48.969007 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:48.968983    2578 patch_prober.go:28] interesting pod/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 10 23:59:48.969061 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:48.969034    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 10 23:59:49.969184 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:49.969135    2578 patch_prober.go:28] interesting pod/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 10 23:59:49.969736 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:49.969207    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 10 23:59:50.968832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:50.968747    2578 patch_prober.go:28] interesting pod/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 10 23:59:50.968832 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:50.968805    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 10 23:59:51.467495 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:51.467453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:51.968917 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:51.968879    2578 patch_prober.go:28] interesting pod/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 10 23:59:51.969131 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:51.968945    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 10 23:59:52.255466 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:52.255348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:52.937117 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:52.937082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:52.969393 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:52.969351    2578 patch_prober.go:28] interesting pod/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 10 23:59:52.969605 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:52.969423    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 10 23:59:53.773143 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:53.773108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:53.972805 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:53.972772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:53.973213 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:53.973123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:53.974002 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:53.973981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 10 23:59:54.463292 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:54.463240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:55.161714 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:55.161677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:56.144043 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:56.144013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:56.619948 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:56.619915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:57.061678 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:57.061646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:57.762935 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:57.762897    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:59.806420 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:59.806383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:59.910939 ip-10-0-128-88 kubenswrapper[2578]: I0910 23:59:59.910905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:00.007031 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.006998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:00.187941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.187845    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"]
Sep 11 00:00:00.191686 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.191656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"
Sep 11 00:00:00.193871 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.193847    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"]
Sep 11 00:00:00.194888 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.194865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\""
Sep 11 00:00:00.197321 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.197302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"
Sep 11 00:00:00.216693 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.216662    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"]
Sep 11 00:00:00.228002 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.227941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"]
Sep 11 00:00:00.236647 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.236615    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"]
Sep 11 00:00:00.239086 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.239044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"
Sep 11 00:00:00.241370 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.241343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"
Sep 11 00:00:00.248042 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.248020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"
Sep 11 00:00:00.261673 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.259643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"]
Sep 11 00:00:00.289904 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.289869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"
Sep 11 00:00:00.542063 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.542011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"]
Sep 11 00:00:00.543830 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:00:00.543789    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bca4e72_0757_4243_ad19_9d7ebbfc4564.slice/crio-83639f83f9b7351bcde5b110ef48b3d02e99cb10a1ddb41dcd98602238e76058 WatchSource:0}: Error finding container 83639f83f9b7351bcde5b110ef48b3d02e99cb10a1ddb41dcd98602238e76058: Status 404 returned error can't find the container with id 83639f83f9b7351bcde5b110ef48b3d02e99cb10a1ddb41dcd98602238e76058
Sep 11 00:00:00.544891 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:00:00.544866    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baf9350_0ee2_4a52_93eb_c822f85870e8.slice/crio-48e2c4abaf747c0cba471e3f4e4dacded23e552eaeb67734138b62787047a462 WatchSource:0}: Error finding container 48e2c4abaf747c0cba471e3f4e4dacded23e552eaeb67734138b62787047a462: Status 404 returned error can't find the container with id 48e2c4abaf747c0cba471e3f4e4dacded23e552eaeb67734138b62787047a462
Sep 11 00:00:00.545198 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.545174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"]
Sep 11 00:00:00.546010 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.545989    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 00:00:00.587911 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:00.587878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"]
Sep 11 00:00:00.588904 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:00:00.588873    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688bd8a0_e5a2_44e9_ab82_92857bdfc4bc.slice/crio-9c13d21c3855c5c771d75718684c0477da0b4ea3b8ecb20c582fe3ddee1ae80e WatchSource:0}: Error finding container 9c13d21c3855c5c771d75718684c0477da0b4ea3b8ecb20c582fe3ddee1ae80e: Status 404 returned error can't find the container with id 9c13d21c3855c5c771d75718684c0477da0b4ea3b8ecb20c582fe3ddee1ae80e
Sep 11 00:00:01.094715 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:01.094679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:01.388262 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:01.388161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs" event={"ID":"5baf9350-0ee2-4a52-93eb-c822f85870e8","Type":"ContainerStarted","Data":"48e2c4abaf747c0cba471e3f4e4dacded23e552eaeb67734138b62787047a462"}
Sep 11 00:00:01.389412 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:01.389392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t" event={"ID":"688bd8a0-e5a2-44e9-ab82-92857bdfc4bc","Type":"ContainerStarted","Data":"9c13d21c3855c5c771d75718684c0477da0b4ea3b8ecb20c582fe3ddee1ae80e"}
Sep 11 00:00:01.390523 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:01.390501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv" event={"ID":"6bca4e72-0757-4243-ad19-9d7ebbfc4564","Type":"ContainerStarted","Data":"83639f83f9b7351bcde5b110ef48b3d02e99cb10a1ddb41dcd98602238e76058"}
Sep 11 00:00:01.978285 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:01.978226    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7" podUID="21b7e23c-f38b-4281-a3d0-81d0992541be" containerName="main" containerID="cri-o://df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e" gracePeriod=45
Sep 11 00:00:02.013955 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.013917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:02.196236 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.196197    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:00:02.302258 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.302221    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 11 00:00:02.399260 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.399177    2578 generic.go:356] "Generic (PLEG): container finished" podID="21b7e23c-f38b-4281-a3d0-81d0992541be" containerID="df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e" exitCode=0
Sep 11 00:00:02.399260 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.399237    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"
Sep 11 00:00:02.399530 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.399239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7" event={"ID":"21b7e23c-f38b-4281-a3d0-81d0992541be","Type":"ContainerDied","Data":"df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e"}
Sep 11 00:00:02.399530 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.399353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7" event={"ID":"21b7e23c-f38b-4281-a3d0-81d0992541be","Type":"ContainerDied","Data":"0cd997d8657845b7e9d307653356bd73647cb19a068c3f8789b1fe6781594c54"}
Sep 11 00:00:02.399530 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.399390    2578 scope.go:117] "RemoveContainer" containerID="df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e"
Sep 11 00:00:02.429542 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.429508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b7e23c-f38b-4281-a3d0-81d0992541be-tls-certs\") pod \"21b7e23c-f38b-4281-a3d0-81d0992541be\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") "
Sep 11 00:00:02.429705 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.429601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6rrj\" (UniqueName: \"kubernetes.io/projected/21b7e23c-f38b-4281-a3d0-81d0992541be-kube-api-access-v6rrj\") pod \"21b7e23c-f38b-4281-a3d0-81d0992541be\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") "
Sep 11 00:00:02.429705 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.429635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-tmp-dir\") pod \"21b7e23c-f38b-4281-a3d0-81d0992541be\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") "
Sep 11 00:00:02.429705 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.429656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-dshm\") pod \"21b7e23c-f38b-4281-a3d0-81d0992541be\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") "
Sep 11 00:00:02.429900 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.429754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-home\") pod \"21b7e23c-f38b-4281-a3d0-81d0992541be\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") "
Sep 11 00:00:02.429900 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.429785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-model-cache\") pod \"21b7e23c-f38b-4281-a3d0-81d0992541be\" (UID: \"21b7e23c-f38b-4281-a3d0-81d0992541be\") "
Sep 11 00:00:02.430369 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.430322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-model-cache" (OuterVolumeSpecName: "model-cache") pod "21b7e23c-f38b-4281-a3d0-81d0992541be" (UID: "21b7e23c-f38b-4281-a3d0-81d0992541be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:02.430369 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.430333    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-home" (OuterVolumeSpecName: "home") pod "21b7e23c-f38b-4281-a3d0-81d0992541be" (UID: "21b7e23c-f38b-4281-a3d0-81d0992541be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:02.430569 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.430380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21b7e23c-f38b-4281-a3d0-81d0992541be" (UID: "21b7e23c-f38b-4281-a3d0-81d0992541be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:02.433788 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.433754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b7e23c-f38b-4281-a3d0-81d0992541be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21b7e23c-f38b-4281-a3d0-81d0992541be" (UID: "21b7e23c-f38b-4281-a3d0-81d0992541be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:02.434015 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.433995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b7e23c-f38b-4281-a3d0-81d0992541be-kube-api-access-v6rrj" (OuterVolumeSpecName: "kube-api-access-v6rrj") pod "21b7e23c-f38b-4281-a3d0-81d0992541be" (UID: "21b7e23c-f38b-4281-a3d0-81d0992541be"). InnerVolumeSpecName "kube-api-access-v6rrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:02.434243 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.434212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-dshm" (OuterVolumeSpecName: "dshm") pod "21b7e23c-f38b-4281-a3d0-81d0992541be" (UID: "21b7e23c-f38b-4281-a3d0-81d0992541be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:02.531014 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.530918    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:02.531014 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.530959    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:02.531014 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.530975    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b7e23c-f38b-4281-a3d0-81d0992541be-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:02.531014 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.530989    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6rrj\" (UniqueName: \"kubernetes.io/projected/21b7e23c-f38b-4281-a3d0-81d0992541be-kube-api-access-v6rrj\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:02.531014 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.531004    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:02.531014 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.531015    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b7e23c-f38b-4281-a3d0-81d0992541be-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:02.698146 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.698103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:02.719913 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.719878    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"]
Sep 11 00:00:02.725452 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:02.725425    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/auth-test-model-6fa41e-kserve-7d89fbcd79-5rkp7"]
Sep 11 00:00:03.092232 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.092187    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8" podUID="2c713a79-2ae0-4a6c-b727-ebc0f20681dc" containerName="main" containerID="cri-o://53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9" gracePeriod=45
Sep 11 00:00:03.096653 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.096629    2578 scope.go:117] "RemoveContainer" containerID="df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e"
Sep 11 00:00:03.097036 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:00:03.097009    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e\": container with ID starting with df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e not found: ID does not exist" containerID="df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e"
Sep 11 00:00:03.097189 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.097044    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e"} err="failed to get container status \"df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e\": rpc error: code = NotFound desc = could not find container \"df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e\": container with ID starting with df3c6d1b1bd02e0c2f0e2270d01e5e95e41ac02e0df088fa906faa55d4fd1d9e not found: ID does not exist"
Sep 11 00:00:03.361450 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.361425    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 11 00:00:03.405268 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.405237    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c713a79-2ae0-4a6c-b727-ebc0f20681dc" containerID="53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9" exitCode=0
Sep 11 00:00:03.405434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.405322    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"
Sep 11 00:00:03.405434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.405324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8" event={"ID":"2c713a79-2ae0-4a6c-b727-ebc0f20681dc","Type":"ContainerDied","Data":"53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9"}
Sep 11 00:00:03.405434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.405368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8" event={"ID":"2c713a79-2ae0-4a6c-b727-ebc0f20681dc","Type":"ContainerDied","Data":"ccbc3665acd6b857fe4f4240c939600e2a2dea45982f2872553d32cb9e60c46f"}
Sep 11 00:00:03.405434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.405391    2578 scope.go:117] "RemoveContainer" containerID="53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9"
Sep 11 00:00:03.413827 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.413810    2578 scope.go:117] "RemoveContainer" containerID="53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9"
Sep 11 00:00:03.414083 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:00:03.414064    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9\": container with ID starting with 53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9 not found: ID does not exist" containerID="53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9"
Sep 11 00:00:03.414145 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.414093    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9"} err="failed to get container status \"53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9\": rpc error: code = NotFound desc = could not find container \"53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9\": container with ID starting with 53bad8fd34875e3d7c9bb0df24263caea4be112debc0af6e549e407c574795c9 not found: ID does not exist"
Sep 11 00:00:03.439565 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tls-certs\") pod \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") "
Sep 11 00:00:03.439679 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-home\") pod \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") "
Sep 11 00:00:03.439679 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb866\" (UniqueName: \"kubernetes.io/projected/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-kube-api-access-pb866\") pod \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") "
Sep 11 00:00:03.439795 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439685    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-model-cache\") pod \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") "
Sep 11 00:00:03.439795 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-dshm\") pod \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") "
Sep 11 00:00:03.439795 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439746    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tmp-dir\") pod \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\" (UID: \"2c713a79-2ae0-4a6c-b727-ebc0f20681dc\") "
Sep 11 00:00:03.440033 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-home" (OuterVolumeSpecName: "home") pod "2c713a79-2ae0-4a6c-b727-ebc0f20681dc" (UID: "2c713a79-2ae0-4a6c-b727-ebc0f20681dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:03.440033 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.439996    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c713a79-2ae0-4a6c-b727-ebc0f20681dc" (UID: "2c713a79-2ae0-4a6c-b727-ebc0f20681dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:03.440160 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.440132    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c713a79-2ae0-4a6c-b727-ebc0f20681dc" (UID: "2c713a79-2ae0-4a6c-b727-ebc0f20681dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:03.440220 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.440162    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:03.440220 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.440183    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:03.442392 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.442349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-dshm" (OuterVolumeSpecName: "dshm") pod "2c713a79-2ae0-4a6c-b727-ebc0f20681dc" (UID: "2c713a79-2ae0-4a6c-b727-ebc0f20681dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:03.442392 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.442373    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c713a79-2ae0-4a6c-b727-ebc0f20681dc" (UID: "2c713a79-2ae0-4a6c-b727-ebc0f20681dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:03.442591 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.442503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-kube-api-access-pb866" (OuterVolumeSpecName: "kube-api-access-pb866") pod "2c713a79-2ae0-4a6c-b727-ebc0f20681dc" (UID: "2c713a79-2ae0-4a6c-b727-ebc0f20681dc"). InnerVolumeSpecName "kube-api-access-pb866". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:03.540954 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.540918    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb866\" (UniqueName: \"kubernetes.io/projected/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-kube-api-access-pb866\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:03.540954 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.540957    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:03.541139 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.540995    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:03.541139 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.541008    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c713a79-2ae0-4a6c-b727-ebc0f20681dc-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:03.729988 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.729954    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"]
Sep 11 00:00:03.734868 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.734841    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/auth-test-model-37c598-kserve-bbbbfcfc9-fbxl8"]
Sep 11 00:00:03.750174 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:03.750146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:04.414164 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.414064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs" event={"ID":"5baf9350-0ee2-4a52-93eb-c822f85870e8","Type":"ContainerStarted","Data":"77fbe90e47e6726b4fd89204545b4f8871370b229aab1437fb356bba691e1da0"}
Sep 11 00:00:04.415688 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.415665    2578 generic.go:356] "Generic (PLEG): container finished" podID="688bd8a0-e5a2-44e9-ab82-92857bdfc4bc" containerID="c16a516bfc7936d93912d4a68ddf8792e06ee376f395dc869abc5ab464b85eba" exitCode=0
Sep 11 00:00:04.415786 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.415734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t" event={"ID":"688bd8a0-e5a2-44e9-ab82-92857bdfc4bc","Type":"ContainerDied","Data":"c16a516bfc7936d93912d4a68ddf8792e06ee376f395dc869abc5ab464b85eba"}
Sep 11 00:00:04.417182 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.417164    2578 generic.go:356] "Generic (PLEG): container finished" podID="6bca4e72-0757-4243-ad19-9d7ebbfc4564" containerID="5fcc10537924dc12c75fa51b6f49c8d892c1642c7bd7094534685e5f7ba8aa43" exitCode=0
Sep 11 00:00:04.417312 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.417228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv" event={"ID":"6bca4e72-0757-4243-ad19-9d7ebbfc4564","Type":"ContainerDied","Data":"5fcc10537924dc12c75fa51b6f49c8d892c1642c7bd7094534685e5f7ba8aa43"}
Sep 11 00:00:04.432125 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.432081    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs" podStartSLOduration=1.759729686 podStartE2EDuration="4.432067305s" totalImagesPullingTime="2.672337619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:04.430000087 +0000 UTC m=+1446.386231887" watchObservedRunningTime="2026-09-11 00:00:04.432067305 +0000 UTC m=+1446.388299108"
Sep 11 00:00:04.465726 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.465685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:04.534567 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.534527    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b7e23c-f38b-4281-a3d0-81d0992541be" path="/var/lib/kubelet/pods/21b7e23c-f38b-4281-a3d0-81d0992541be/volumes"
Sep 11 00:00:04.534930 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.534917    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c713a79-2ae0-4a6c-b727-ebc0f20681dc" path="/var/lib/kubelet/pods/2c713a79-2ae0-4a6c-b727-ebc0f20681dc/volumes"
Sep 11 00:00:04.638935 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:04.638903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:05.011375 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.011346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:05.111207 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.111165    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"]
Sep 11 00:00:05.111632 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.111617    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c713a79-2ae0-4a6c-b727-ebc0f20681dc" containerName="main"
Sep 11 00:00:05.111699 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.111634    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c713a79-2ae0-4a6c-b727-ebc0f20681dc" containerName="main"
Sep 11 00:00:05.111699 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.111649    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b7e23c-f38b-4281-a3d0-81d0992541be" containerName="main"
Sep 11 00:00:05.111699 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.111655    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b7e23c-f38b-4281-a3d0-81d0992541be" containerName="main"
Sep 11 00:00:05.111816 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.111727    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c713a79-2ae0-4a6c-b727-ebc0f20681dc" containerName="main"
Sep 11 00:00:05.111816 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.111733    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b7e23c-f38b-4281-a3d0-81d0992541be" containerName="main"
Sep 11 00:00:05.114875 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.114854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.117806 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.117781    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-02bac9\""
Sep 11 00:00:05.117932 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.117847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-02bac9-dockercfg-862hs\""
Sep 11 00:00:05.126829 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.126804    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"]
Sep 11 00:00:05.171215 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.171179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:05.226758 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.226726    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"]
Sep 11 00:00:05.230446 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.230424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.241780 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.241756    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"]
Sep 11 00:00:05.260235 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.260185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvhd\" (UniqueName: \"kubernetes.io/projected/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-kube-api-access-4lvhd\") pod \"payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.260375 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.260319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.360926 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.360891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvhd\" (UniqueName: \"kubernetes.io/projected/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-kube-api-access-4lvhd\") pod \"payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.361090 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.360941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztz2j\" (UniqueName: \"kubernetes.io/projected/f90f1b9b-6254-47c4-a44e-41ea43b10f62-kube-api-access-ztz2j\") pod \"payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") " pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.361090 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.360975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.361090 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.360999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f1b9b-6254-47c4-a44e-41ea43b10f62-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") " pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.361702 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.361683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.369459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.369436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvhd\" (UniqueName: \"kubernetes.io/projected/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-kube-api-access-4lvhd\") pod \"payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.424538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.424501    2578 generic.go:356] "Generic (PLEG): container finished" podID="5baf9350-0ee2-4a52-93eb-c822f85870e8" containerID="77fbe90e47e6726b4fd89204545b4f8871370b229aab1437fb356bba691e1da0" exitCode=0
Sep 11 00:00:05.424971 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.424589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs" event={"ID":"5baf9350-0ee2-4a52-93eb-c822f85870e8","Type":"ContainerDied","Data":"77fbe90e47e6726b4fd89204545b4f8871370b229aab1437fb356bba691e1da0"}
Sep 11 00:00:05.427181 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.427156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:05.462356 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.461986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztz2j\" (UniqueName: \"kubernetes.io/projected/f90f1b9b-6254-47c4-a44e-41ea43b10f62-kube-api-access-ztz2j\") pod \"payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") " pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.462356 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.462078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f1b9b-6254-47c4-a44e-41ea43b10f62-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") " pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.463203 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.463144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f1b9b-6254-47c4-a44e-41ea43b10f62-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") " pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.472438 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.472350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztz2j\" (UniqueName: \"kubernetes.io/projected/f90f1b9b-6254-47c4-a44e-41ea43b10f62-kube-api-access-ztz2j\") pod \"payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") " pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.543737 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.543707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:05.582020 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.581975    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"
Sep 11 00:00:05.621027 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.620985    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"
Sep 11 00:00:05.712725 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.712702    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"]
Sep 11 00:00:05.714728 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:00:05.714699    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc0ca52_68c9_41f1_b250_cbe0ec92e57d.slice/crio-a450952695f3f99ef05268ca15632002d8d7062951e935fa22d4a79a582378cc WatchSource:0}: Error finding container a450952695f3f99ef05268ca15632002d8d7062951e935fa22d4a79a582378cc: Status 404 returned error can't find the container with id a450952695f3f99ef05268ca15632002d8d7062951e935fa22d4a79a582378cc
Sep 11 00:00:05.833231 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:05.833190    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"]
Sep 11 00:00:05.836584 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:00:05.836143    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f1b9b_6254_47c4_a44e_41ea43b10f62.slice/crio-101600afd62fa6c68a54e44f79b607b47e3bf8fbf68cd54b7fe65efa020894bf WatchSource:0}: Error finding container 101600afd62fa6c68a54e44f79b607b47e3bf8fbf68cd54b7fe65efa020894bf: Status 404 returned error can't find the container with id 101600afd62fa6c68a54e44f79b607b47e3bf8fbf68cd54b7fe65efa020894bf
Sep 11 00:00:06.431517 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.431481    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"
Sep 11 00:00:06.431517 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.431493    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t" event={"ID":"688bd8a0-e5a2-44e9-ab82-92857bdfc4bc","Type":"ContainerDied","Data":"9c13d21c3855c5c771d75718684c0477da0b4ea3b8ecb20c582fe3ddee1ae80e"}
Sep 11 00:00:06.432045 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.431582    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c13d21c3855c5c771d75718684c0477da0b4ea3b8ecb20c582fe3ddee1ae80e"
Sep 11 00:00:06.433442 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.433151    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"
Sep 11 00:00:06.433442 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.433151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv" event={"ID":"6bca4e72-0757-4243-ad19-9d7ebbfc4564","Type":"ContainerDied","Data":"83639f83f9b7351bcde5b110ef48b3d02e99cb10a1ddb41dcd98602238e76058"}
Sep 11 00:00:06.433442 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.433308    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83639f83f9b7351bcde5b110ef48b3d02e99cb10a1ddb41dcd98602238e76058"
Sep 11 00:00:06.435116 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.435092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf" event={"ID":"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d","Type":"ContainerStarted","Data":"6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292"}
Sep 11 00:00:06.435226 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.435143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf" event={"ID":"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d","Type":"ContainerStarted","Data":"a450952695f3f99ef05268ca15632002d8d7062951e935fa22d4a79a582378cc"}
Sep 11 00:00:06.435226 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.435198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:06.436880 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.436844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk" event={"ID":"f90f1b9b-6254-47c4-a44e-41ea43b10f62","Type":"ContainerStarted","Data":"ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a"}
Sep 11 00:00:06.437058 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.437042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:06.437163 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.437151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk" event={"ID":"f90f1b9b-6254-47c4-a44e-41ea43b10f62","Type":"ContainerStarted","Data":"101600afd62fa6c68a54e44f79b607b47e3bf8fbf68cd54b7fe65efa020894bf"}
Sep 11 00:00:06.457211 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.457155    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf" podStartSLOduration=1.457133249 podStartE2EDuration="1.457133249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:06.45335654 +0000 UTC m=+1448.409588344" watchObservedRunningTime="2026-09-11 00:00:06.457133249 +0000 UTC m=+1448.413365053"
Sep 11 00:00:06.585532 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.585500    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"
Sep 11 00:00:06.602571 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.602513    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk" podStartSLOduration=1.6024919789999998 podStartE2EDuration="1.602491979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:06.475022601 +0000 UTC m=+1448.431254406" watchObservedRunningTime="2026-09-11 00:00:06.602491979 +0000 UTC m=+1448.558723782"
Sep 11 00:00:06.725487 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.725394    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"]
Sep 11 00:00:06.725910 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.725891    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf9350-0ee2-4a52-93eb-c822f85870e8" containerName="cleanup"
Sep 11 00:00:06.726003 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.725914    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf9350-0ee2-4a52-93eb-c822f85870e8" containerName="cleanup"
Sep 11 00:00:06.726003 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.725934    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688bd8a0-e5a2-44e9-ab82-92857bdfc4bc" containerName="cleanup"
Sep 11 00:00:06.726003 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.725942    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="688bd8a0-e5a2-44e9-ab82-92857bdfc4bc" containerName="cleanup"
Sep 11 00:00:06.726003 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.725963    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bca4e72-0757-4243-ad19-9d7ebbfc4564" containerName="cleanup"
Sep 11 00:00:06.726003 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.725971    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bca4e72-0757-4243-ad19-9d7ebbfc4564" containerName="cleanup"
Sep 11 00:00:06.726248 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.726066    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baf9350-0ee2-4a52-93eb-c822f85870e8" containerName="cleanup"
Sep 11 00:00:06.726248 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.726083    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="688bd8a0-e5a2-44e9-ab82-92857bdfc4bc" containerName="cleanup"
Sep 11 00:00:06.726248 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.726093    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bca4e72-0757-4243-ad19-9d7ebbfc4564" containerName="cleanup"
Sep 11 00:00:06.729137 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.729116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.732150 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.731958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"kube-root-ca.crt\""
Sep 11 00:00:06.732150 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.732007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"openshift-service-ca.crt\""
Sep 11 00:00:06.732150 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.732058    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"sub-test-model-6fa41e-kserve-self-signed-certs\""
Sep 11 00:00:06.732401 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.732364    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"default-dockercfg-rnn8j\""
Sep 11 00:00:06.739831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.739801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"]
Sep 11 00:00:06.840810 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.840782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:06.877744 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.877705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tls-certs\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.877929 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.877774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-model-cache\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.877929 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.877801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-dshm\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.877929 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.877819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tmp-dir\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.877929 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.877848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb8tm\" (UniqueName: \"kubernetes.io/projected/d6102ddf-5bed-4f4c-bad3-4249df44b31f-kube-api-access-bb8tm\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.878122 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.877924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-home\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.979609 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.979508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tls-certs\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.979609 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.979584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-model-cache\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.979854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.979613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-dshm\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.979854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.979640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tmp-dir\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.979854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.979674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb8tm\" (UniqueName: \"kubernetes.io/projected/d6102ddf-5bed-4f4c-bad3-4249df44b31f-kube-api-access-bb8tm\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.979854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.979730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-home\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.980120 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.980087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-model-cache\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.980120 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.980103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-home\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.980352 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.980142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tmp-dir\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.981966 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.981936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-dshm\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.982534 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.982510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tls-certs\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:06.988869 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:06.988844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb8tm\" (UniqueName: \"kubernetes.io/projected/d6102ddf-5bed-4f4c-bad3-4249df44b31f-kube-api-access-bb8tm\") pod \"sub-test-model-6fa41e-kserve-8574dc6569-d7z64\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:07.040539 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.040501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:07.311591 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.311556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"]
Sep 11 00:00:07.313384 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:00:07.313354    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6102ddf_5bed_4f4c_bad3_4249df44b31f.slice/crio-411e39736aee0c589bd37ae8bca62830696176e938d1d7b73ebcff5d6a6e246e WatchSource:0}: Error finding container 411e39736aee0c589bd37ae8bca62830696176e938d1d7b73ebcff5d6a6e246e: Status 404 returned error can't find the container with id 411e39736aee0c589bd37ae8bca62830696176e938d1d7b73ebcff5d6a6e246e
Sep 11 00:00:07.444214 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.444182    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"
Sep 11 00:00:07.444725 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.444211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs" event={"ID":"5baf9350-0ee2-4a52-93eb-c822f85870e8","Type":"ContainerDied","Data":"48e2c4abaf747c0cba471e3f4e4dacded23e552eaeb67734138b62787047a462"}
Sep 11 00:00:07.444725 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.444265    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e2c4abaf747c0cba471e3f4e4dacded23e552eaeb67734138b62787047a462"
Sep 11 00:00:07.445993 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.445965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64" event={"ID":"d6102ddf-5bed-4f4c-bad3-4249df44b31f","Type":"ContainerStarted","Data":"247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c"}
Sep 11 00:00:07.446141 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.446016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64" event={"ID":"d6102ddf-5bed-4f4c-bad3-4249df44b31f","Type":"ContainerStarted","Data":"411e39736aee0c589bd37ae8bca62830696176e938d1d7b73ebcff5d6a6e246e"}
Sep 11 00:00:07.463499 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:07.463453    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64" podStartSLOduration=1.463437682 podStartE2EDuration="1.463437682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:07.462417601 +0000 UTC m=+1449.418649405" watchObservedRunningTime="2026-09-11 00:00:07.463437682 +0000 UTC m=+1449.419669484"
Sep 11 00:00:17.041123 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:17.041081    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:17.041642 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:17.041142    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:17.053771 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:17.053746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:17.448295 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:17.448197    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:00:17.448432 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:17.448322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:00:17.515492 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:17.515459    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:00:18.587750 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:18.587715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:00:26.766087 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:26.766044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:27.591840 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:27.591761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:28.383790 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:28.383751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:29.177176 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:29.177134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:30.004930 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.004884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:30.105180 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.105144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:30.747074 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.747038    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"]
Sep 11 00:00:30.750941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.750921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.754760 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.754739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"default-dockercfg-rxxx8\""
Sep 11 00:00:30.754760 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.754749    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"sub-test-model-37c598-kserve-self-signed-certs\""
Sep 11 00:00:30.754945 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.754751    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"kube-root-ca.crt\""
Sep 11 00:00:30.754945 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.754751    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"openshift-service-ca.crt\""
Sep 11 00:00:30.759388 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.759366    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"]
Sep 11 00:00:30.795207 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.795170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tls-certs\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.795392 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.795287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-dshm\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.795446 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.795389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tmp-dir\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.795490 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.795474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46gk\" (UniqueName: \"kubernetes.io/projected/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-kube-api-access-f46gk\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.795528 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.795512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-model-cache\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.795572 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.795538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-home\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.845141 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.845109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:30.896443 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-model-cache\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.896634 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-home\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.896634 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tls-certs\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.896762 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-dshm\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.896831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tmp-dir\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.896892 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-model-cache\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.896945 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-home\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.896945 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.896907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46gk\" (UniqueName: \"kubernetes.io/projected/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-kube-api-access-f46gk\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.897117 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.897096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tmp-dir\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.898992 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.898959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-dshm\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.899110 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.899095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tls-certs\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.905511 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.905491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46gk\" (UniqueName: \"kubernetes.io/projected/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-kube-api-access-f46gk\") pod \"sub-test-model-37c598-kserve-688bdc94cc-nztck\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") " pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:30.950613 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:30.950579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:31.061674 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:31.061639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:31.333604 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:31.333572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"]
Sep 11 00:00:31.335424 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:00:31.335399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3161a16_1d6e_4c82_a2c6_7f52a2f3d744.slice/crio-2f494900ad2531db4bb5da07d6df4a1c5f10da54b2c95e8811bd8d2a6d4d4d3e WatchSource:0}: Error finding container 2f494900ad2531db4bb5da07d6df4a1c5f10da54b2c95e8811bd8d2a6d4d4d3e: Status 404 returned error can't find the container with id 2f494900ad2531db4bb5da07d6df4a1c5f10da54b2c95e8811bd8d2a6d4d4d3e
Sep 11 00:00:31.579534 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:31.579494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck" event={"ID":"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744","Type":"ContainerStarted","Data":"fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e"}
Sep 11 00:00:31.579722 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:31.579552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck" event={"ID":"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744","Type":"ContainerStarted","Data":"2f494900ad2531db4bb5da07d6df4a1c5f10da54b2c95e8811bd8d2a6d4d4d3e"}
Sep 11 00:00:31.600749 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:31.600654    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck" podStartSLOduration=1.600639232 podStartE2EDuration="1.600639232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:31.599494199 +0000 UTC m=+1473.555726001" watchObservedRunningTime="2026-09-11 00:00:31.600639232 +0000 UTC m=+1473.556871075"
Sep 11 00:00:32.122325 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:32.122286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:33.288124 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:33.288090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:34.258523 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:34.258489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:34.982068 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:34.982037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:35.872534 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:35.872499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:35.994949 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:35.994914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:00:36.679025 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:36.678975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:37.385375 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:37.385339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:38.353295 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:38.353245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:39.181231 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:39.181196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:41.061886 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:41.061845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:41.062319 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:41.061906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:41.074543 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:41.074515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:41.642477 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:41.642440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:00:48.592116 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.592082    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"]
Sep 11 00:00:48.594714 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.594691    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-afe559-29818080-qnz7t"]
Sep 11 00:00:48.597922 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.597895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:48.619998 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.619973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"]
Sep 11 00:00:48.620383 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.620350    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4" podUID="22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" containerName="payload-processing" containerID="cri-o://918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb" gracePeriod=30
Sep 11 00:00:48.666113 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.666083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:48.767048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.767010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:48.824744 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.824716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:48.877760 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.877724    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 11 00:00:48.972635 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.972607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-plugins-config-volume\") pod \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") "
Sep 11 00:00:48.972777 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.972705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwr6c\" (UniqueName: \"kubernetes.io/projected/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-kube-api-access-mwr6c\") pod \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\" (UID: \"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed\") "
Sep 11 00:00:48.973050 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.973019    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" (UID: "22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:48.974682 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:48.974659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-kube-api-access-mwr6c" (OuterVolumeSpecName: "kube-api-access-mwr6c") pod "22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" (UID: "22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed"). InnerVolumeSpecName "kube-api-access-mwr6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:49.073676 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.073650    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:49.073676 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.073672    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwr6c\" (UniqueName: \"kubernetes.io/projected/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed-kube-api-access-mwr6c\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:00:49.156761 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.156671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:49.292578 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.292549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:49.674881 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.674841    2578 generic.go:356] "Generic (PLEG): container finished" podID="22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" containerID="918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb" exitCode=0
Sep 11 00:00:49.675454 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.674895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"
Sep 11 00:00:49.675454 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.674911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4" event={"ID":"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed","Type":"ContainerDied","Data":"918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb"}
Sep 11 00:00:49.675454 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.674991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4" event={"ID":"22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed","Type":"ContainerDied","Data":"cc7810ed0d8f377c77c45632dba81d268eb599d9f7b51261744d567006e1eecc"}
Sep 11 00:00:49.675454 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.675027    2578 scope.go:117] "RemoveContainer" containerID="918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb"
Sep 11 00:00:49.684087 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.684070    2578 scope.go:117] "RemoveContainer" containerID="918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb"
Sep 11 00:00:49.684371 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:00:49.684350    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb\": container with ID starting with 918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb not found: ID does not exist" containerID="918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb"
Sep 11 00:00:49.684442 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.684381    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb"} err="failed to get container status \"918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb\": rpc error: code = NotFound desc = could not find container \"918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb\": container with ID starting with 918e75f9ac3c1cfe1baec2fee4401075411c1281700b6733705caebe10d01fcb not found: ID does not exist"
Sep 11 00:00:49.697503 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.697479    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"]
Sep 11 00:00:49.699482 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:49.699463    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-afe559-784f548b77-tkrz4"]
Sep 11 00:00:50.098386 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:50.098354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:50.534831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:50.534795    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" path="/var/lib/kubelet/pods/22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed/volumes"
Sep 11 00:00:50.535104 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:50.535090    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688bd8a0-e5a2-44e9-ab82-92857bdfc4bc" path="/var/lib/kubelet/pods/688bd8a0-e5a2-44e9-ab82-92857bdfc4bc/volumes"
Sep 11 00:00:50.876641 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:50.876549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:51.570829 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:51.570791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:52.257699 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:52.257667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:53.083928 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:53.083897    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:54.894215 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:54.894177    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:54.993368 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:54.993333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.100066 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:55.100032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:00:55.154939 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:55.154848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"]
Sep 11 00:00:55.155635 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:55.155580    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy" containerID="cri-o://2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798" gracePeriod=30
Sep 11 00:00:55.181318 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:55.181264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.696780 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:55.696745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.886374 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:55.886335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:56.864771 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:56.864734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:57.797471 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:57.797429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:58.586509 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:58.586473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:58.617523 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:58.617495    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 11 00:00:58.621358 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:58.621334    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 11 00:00:59.567409 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:00:59.567369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:00.425165 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.425102    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 11 00:01:00.497003 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.496960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:00.585248 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585205    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxff9\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-kube-api-access-rxff9\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585470 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-istio-token\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585470 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-socket\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585470 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334c6b48-0d1d-414e-9117-4c401741c484-istio-podinfo\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585470 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-envoy\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585470 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334c6b48-0d1d-414e-9117-4c401741c484-istiod-ca-cert\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585771 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585479    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-certs\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585771 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585520    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-credential-socket\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585771 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585546    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-data\") pod \"334c6b48-0d1d-414e-9117-4c401741c484\" (UID: \"334c6b48-0d1d-414e-9117-4c401741c484\") "
Sep 11 00:01:00.585771 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:00.585976 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:00.585976 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585903    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.585976 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585924    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-workload-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.585976 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585923    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-data" (OuterVolumeSpecName: "istio-data") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:00.585976 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.585962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334c6b48-0d1d-414e-9117-4c401741c484-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:01:00.586251 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.586002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:00.587551 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.587519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/334c6b48-0d1d-414e-9117-4c401741c484-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 00:01:00.587692 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.587571    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-kube-api-access-rxff9" (OuterVolumeSpecName: "kube-api-access-rxff9") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "kube-api-access-rxff9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:01:00.587692 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.587676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-istio-token" (OuterVolumeSpecName: "istio-token") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:01:00.587947 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.587928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "334c6b48-0d1d-414e-9117-4c401741c484" (UID: "334c6b48-0d1d-414e-9117-4c401741c484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:00.687252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.687149    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-credential-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.687252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.687186    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-data\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.687252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.687200    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxff9\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-kube-api-access-rxff9\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.687252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.687212    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334c6b48-0d1d-414e-9117-4c401741c484-istio-token\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.687252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.687224    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334c6b48-0d1d-414e-9117-4c401741c484-istio-podinfo\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.687252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.687234    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334c6b48-0d1d-414e-9117-4c401741c484-istio-envoy\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.687252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.687245    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334c6b48-0d1d-414e-9117-4c401741c484-istiod-ca-cert\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:00.735115 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.735085    2578 generic.go:356] "Generic (PLEG): container finished" podID="334c6b48-0d1d-414e-9117-4c401741c484" containerID="2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798" exitCode=0
Sep 11 00:01:00.735310 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.735138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"
Sep 11 00:01:00.735310 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.735165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" event={"ID":"334c6b48-0d1d-414e-9117-4c401741c484","Type":"ContainerDied","Data":"2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798"}
Sep 11 00:01:00.735310 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.735230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd" event={"ID":"334c6b48-0d1d-414e-9117-4c401741c484","Type":"ContainerDied","Data":"8899ba7a15d78ebb2133434f64d6ed3619f6271266d03ae4d88b86ecd76e4213"}
Sep 11 00:01:00.735310 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.735258    2578 scope.go:117] "RemoveContainer" containerID="2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798"
Sep 11 00:01:00.746431 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.746410    2578 scope.go:117] "RemoveContainer" containerID="17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80"
Sep 11 00:01:00.754694 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.754621    2578 scope.go:117] "RemoveContainer" containerID="2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798"
Sep 11 00:01:00.756515 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:01:00.756383    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798\": container with ID starting with 2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798 not found: ID does not exist" containerID="2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798"
Sep 11 00:01:00.756515 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.756423    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798"} err="failed to get container status \"2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798\": rpc error: code = NotFound desc = could not find container \"2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798\": container with ID starting with 2ff4946a23dae5cb3decc153796ee0ba0c4559b79d9f880732003df7466d7798 not found: ID does not exist"
Sep 11 00:01:00.756515 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.756447    2578 scope.go:117] "RemoveContainer" containerID="17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80"
Sep 11 00:01:00.756824 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:01:00.756771    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80\": container with ID starting with 17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80 not found: ID does not exist" containerID="17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80"
Sep 11 00:01:00.756824 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.756797    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80"} err="failed to get container status \"17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80\": rpc error: code = NotFound desc = could not find container \"17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80\": container with ID starting with 17de327b3688986408f749a97a7b7989fee6241cbb9f9462e1085ed1ef64fb80 not found: ID does not exist"
Sep 11 00:01:00.760066 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.760042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"]
Sep 11 00:01:00.762979 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:00.762957    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-afe559-openshift-default-747cbf7cc4-hn4xd"]
Sep 11 00:01:01.513765 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:01.513728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:02.535719 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:02.535683    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334c6b48-0d1d-414e-9117-4c401741c484" path="/var/lib/kubelet/pods/334c6b48-0d1d-414e-9117-4c401741c484/volumes"
Sep 11 00:01:06.680630 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:06.680593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:11.038093 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.038051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:11.066267 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.066234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"]
Sep 11 00:01:11.066634 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.066602    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk" podUID="f90f1b9b-6254-47c4-a44e-41ea43b10f62" containerName="payload-processing" containerID="cri-o://ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a" gracePeriod=30
Sep 11 00:01:11.072702 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.072676    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"]
Sep 11 00:01:11.072898 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.072875    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf" podUID="2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" containerName="payload-pre-processing" containerID="cri-o://6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292" gracePeriod=30
Sep 11 00:01:11.106960 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.106921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:11.213014 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.212963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:11.314030 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.313933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:11.353252 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.353224    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:01:11.356506 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.356481    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:01:11.382649 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.382613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lvhd\" (UniqueName: \"kubernetes.io/projected/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-kube-api-access-4lvhd\") pod \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") "
Sep 11 00:01:11.382809 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.382671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f1b9b-6254-47c4-a44e-41ea43b10f62-plugins-config-volume\") pod \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") "
Sep 11 00:01:11.382809 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.382706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztz2j\" (UniqueName: \"kubernetes.io/projected/f90f1b9b-6254-47c4-a44e-41ea43b10f62-kube-api-access-ztz2j\") pod \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\" (UID: \"f90f1b9b-6254-47c4-a44e-41ea43b10f62\") "
Sep 11 00:01:11.382809 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.382738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-plugins-config-volume\") pod \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\" (UID: \"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d\") "
Sep 11 00:01:11.383422 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.383387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90f1b9b-6254-47c4-a44e-41ea43b10f62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f90f1b9b-6254-47c4-a44e-41ea43b10f62" (UID: "f90f1b9b-6254-47c4-a44e-41ea43b10f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:01:11.383584 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.383393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" (UID: "2cc0ca52-68c9-41f1-b250-cbe0ec92e57d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:01:11.385901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.385872    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90f1b9b-6254-47c4-a44e-41ea43b10f62-kube-api-access-ztz2j" (OuterVolumeSpecName: "kube-api-access-ztz2j") pod "f90f1b9b-6254-47c4-a44e-41ea43b10f62" (UID: "f90f1b9b-6254-47c4-a44e-41ea43b10f62"). InnerVolumeSpecName "kube-api-access-ztz2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:01:11.386028 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.386012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-kube-api-access-4lvhd" (OuterVolumeSpecName: "kube-api-access-4lvhd") pod "2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" (UID: "2cc0ca52-68c9-41f1-b250-cbe0ec92e57d"). InnerVolumeSpecName "kube-api-access-4lvhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:01:11.483994 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.483953    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lvhd\" (UniqueName: \"kubernetes.io/projected/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-kube-api-access-4lvhd\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:11.483994 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.483984    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f1b9b-6254-47c4-a44e-41ea43b10f62-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:11.483994 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.483994    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztz2j\" (UniqueName: \"kubernetes.io/projected/f90f1b9b-6254-47c4-a44e-41ea43b10f62-kube-api-access-ztz2j\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:11.483994 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.484002    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:11.802555 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.802520    2578 generic.go:356] "Generic (PLEG): container finished" podID="2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" containerID="6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292" exitCode=0
Sep 11 00:01:11.802738 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.802596    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"
Sep 11 00:01:11.802738 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.802610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf" event={"ID":"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d","Type":"ContainerDied","Data":"6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292"}
Sep 11 00:01:11.802738 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.802677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf" event={"ID":"2cc0ca52-68c9-41f1-b250-cbe0ec92e57d","Type":"ContainerDied","Data":"a450952695f3f99ef05268ca15632002d8d7062951e935fa22d4a79a582378cc"}
Sep 11 00:01:11.802738 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.802708    2578 scope.go:117] "RemoveContainer" containerID="6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292"
Sep 11 00:01:11.804011 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.803982    2578 generic.go:356] "Generic (PLEG): container finished" podID="f90f1b9b-6254-47c4-a44e-41ea43b10f62" containerID="ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a" exitCode=0
Sep 11 00:01:11.804123 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.804013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk" event={"ID":"f90f1b9b-6254-47c4-a44e-41ea43b10f62","Type":"ContainerDied","Data":"ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a"}
Sep 11 00:01:11.804123 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.804046    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"
Sep 11 00:01:11.804123 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.804063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk" event={"ID":"f90f1b9b-6254-47c4-a44e-41ea43b10f62","Type":"ContainerDied","Data":"101600afd62fa6c68a54e44f79b607b47e3bf8fbf68cd54b7fe65efa020894bf"}
Sep 11 00:01:11.811786 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.811723    2578 scope.go:117] "RemoveContainer" containerID="6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292"
Sep 11 00:01:11.812077 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:01:11.812055    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292\": container with ID starting with 6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292 not found: ID does not exist" containerID="6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292"
Sep 11 00:01:11.812149 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.812085    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292"} err="failed to get container status \"6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292\": rpc error: code = NotFound desc = could not find container \"6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292\": container with ID starting with 6f4d8c003d9d77c1720dc206126c053d52bd7ff73417bf172402bbb8fcac4292 not found: ID does not exist"
Sep 11 00:01:11.812149 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.812112    2578 scope.go:117] "RemoveContainer" containerID="ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a"
Sep 11 00:01:11.820482 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.820464    2578 scope.go:117] "RemoveContainer" containerID="ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a"
Sep 11 00:01:11.820761 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:01:11.820740    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a\": container with ID starting with ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a not found: ID does not exist" containerID="ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a"
Sep 11 00:01:11.820811 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.820768    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a"} err="failed to get container status \"ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a\": rpc error: code = NotFound desc = could not find container \"ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a\": container with ID starting with ee5ce93d4bbbbba223b9615221a4fbe15d38b72d507c8cb31dcbfe39f65da02a not found: ID does not exist"
Sep 11 00:01:11.827954 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.827929    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"]
Sep 11 00:01:11.831490 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.831463    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02bac9-647b78cb86-jbwdf"]
Sep 11 00:01:11.841571 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.841536    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"]
Sep 11 00:01:11.843680 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:11.843657    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02bac9-6d56fc6c9-hhrsk"]
Sep 11 00:01:12.216394 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:12.216294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:12.534259 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:12.534227    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" path="/var/lib/kubelet/pods/2cc0ca52-68c9-41f1-b250-cbe0ec92e57d/volumes"
Sep 11 00:01:12.534683 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:12.534667    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90f1b9b-6254-47c4-a44e-41ea43b10f62" path="/var/lib/kubelet/pods/f90f1b9b-6254-47c4-a44e-41ea43b10f62/volumes"
Sep 11 00:01:12.552383 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:12.552357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:12.652815 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:12.652772    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:13.271179 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:13.271144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:14.097551 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:14.097519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:14.593588 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:14.593556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:01:14.834738 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:14.834702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:17.255231 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:17.255190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:17.356317 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:17.356258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:17.531147 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:17.530555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:18.334672 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:18.334631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:20.504859 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:20.504821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:20.571598 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:20.571563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:20.660408 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:20.660369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:21.367134 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:21.367045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:22.064452 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:22.064415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:24.473432 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:24.473395    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"]
Sep 11 00:01:24.497441 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:24.497412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:25.618548 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:25.618513    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"]
Sep 11 00:01:25.637102 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:25.637069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:28.396148 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.396113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:28.420367 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.420331    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"]
Sep 11 00:01:28.420630 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.420605    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7" podUID="859f5875-2761-4211-b221-9a91ab9d770f" containerName="payload-processing" containerID="cri-o://fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab" gracePeriod=30
Sep 11 00:01:28.496369 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.496339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:28.590353 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.590319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:28.655062 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.655031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:28.674473 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.674448    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 11 00:01:28.744737 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.744712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859f5875-2761-4211-b221-9a91ab9d770f-plugins-config-volume\") pod \"859f5875-2761-4211-b221-9a91ab9d770f\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") "
Sep 11 00:01:28.744896 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.744745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b68tx\" (UniqueName: \"kubernetes.io/projected/859f5875-2761-4211-b221-9a91ab9d770f-kube-api-access-b68tx\") pod \"859f5875-2761-4211-b221-9a91ab9d770f\" (UID: \"859f5875-2761-4211-b221-9a91ab9d770f\") "
Sep 11 00:01:28.745067 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.745043    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859f5875-2761-4211-b221-9a91ab9d770f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "859f5875-2761-4211-b221-9a91ab9d770f" (UID: "859f5875-2761-4211-b221-9a91ab9d770f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:01:28.746976 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.746950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859f5875-2761-4211-b221-9a91ab9d770f-kube-api-access-b68tx" (OuterVolumeSpecName: "kube-api-access-b68tx") pod "859f5875-2761-4211-b221-9a91ab9d770f" (UID: "859f5875-2761-4211-b221-9a91ab9d770f"). InnerVolumeSpecName "kube-api-access-b68tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:01:28.845884 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.845853    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859f5875-2761-4211-b221-9a91ab9d770f-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:28.845884 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.845878    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b68tx\" (UniqueName: \"kubernetes.io/projected/859f5875-2761-4211-b221-9a91ab9d770f-kube-api-access-b68tx\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:28.900408 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.900377    2578 generic.go:356] "Generic (PLEG): container finished" podID="859f5875-2761-4211-b221-9a91ab9d770f" containerID="fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab" exitCode=0
Sep 11 00:01:28.900531 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.900422    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"
Sep 11 00:01:28.900531 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.900455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7" event={"ID":"859f5875-2761-4211-b221-9a91ab9d770f","Type":"ContainerDied","Data":"fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab"}
Sep 11 00:01:28.900531 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.900506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7" event={"ID":"859f5875-2761-4211-b221-9a91ab9d770f","Type":"ContainerDied","Data":"68a287c6ec0ebf878fdfdd540d0e8a7287e23d753efc660735da1117078c1654"}
Sep 11 00:01:28.900657 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.900536    2578 scope.go:117] "RemoveContainer" containerID="fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab"
Sep 11 00:01:28.909849 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.909833    2578 scope.go:117] "RemoveContainer" containerID="fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab"
Sep 11 00:01:28.910109 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:01:28.910087    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab\": container with ID starting with fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab not found: ID does not exist" containerID="fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab"
Sep 11 00:01:28.910188 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.910120    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab"} err="failed to get container status \"fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab\": rpc error: code = NotFound desc = could not find container \"fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab\": container with ID starting with fa227c5acb887b4ae74e34ce57e78a27f5ea93306b6e86d78ce45b933ce591ab not found: ID does not exist"
Sep 11 00:01:28.939925 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.939900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"]
Sep 11 00:01:28.952409 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:28.952383    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396319-6b7f6dcd59-672h7"]
Sep 11 00:01:29.010158 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:29.010129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:29.144895 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:29.144864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:29.288922 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:29.288888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:01:30.534376 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:30.534343    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859f5875-2761-4211-b221-9a91ab9d770f" path="/var/lib/kubelet/pods/859f5875-2761-4211-b221-9a91ab9d770f/volumes"
Sep 11 00:01:30.546084 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:30.546051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:30.758750 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:30.758708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:31.570301 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:31.570257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:34.593945 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:34.593911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:34.660644 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:34.660612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:34.854897 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:34.854809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:35.428989 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:35.428947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:36.176426 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:36.176394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:36.865930 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:36.865894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:37.291219 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:37.291188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:37.669553 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:37.669467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:38.416410 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:38.416372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:39.494572 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.494529    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64" podUID="d6102ddf-5bed-4f4c-bad3-4249df44b31f" containerName="main" containerID="cri-o://247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c" gracePeriod=45
Sep 11 00:01:39.745768 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.745702    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:01:39.852298 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-model-cache\") pod \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") "
Sep 11 00:01:39.852477 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tls-certs\") pod \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") "
Sep 11 00:01:39.852477 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb8tm\" (UniqueName: \"kubernetes.io/projected/d6102ddf-5bed-4f4c-bad3-4249df44b31f-kube-api-access-bb8tm\") pod \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") "
Sep 11 00:01:39.852477 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852398    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-dshm\") pod \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") "
Sep 11 00:01:39.852477 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852432    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tmp-dir\") pod \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") "
Sep 11 00:01:39.852687 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-home\") pod \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\" (UID: \"d6102ddf-5bed-4f4c-bad3-4249df44b31f\") "
Sep 11 00:01:39.852687 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6102ddf-5bed-4f4c-bad3-4249df44b31f" (UID: "d6102ddf-5bed-4f4c-bad3-4249df44b31f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:39.852799 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6102ddf-5bed-4f4c-bad3-4249df44b31f" (UID: "d6102ddf-5bed-4f4c-bad3-4249df44b31f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:39.852799 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852787    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:39.852901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852821    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:39.852901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.852810    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-home" (OuterVolumeSpecName: "home") pod "d6102ddf-5bed-4f4c-bad3-4249df44b31f" (UID: "d6102ddf-5bed-4f4c-bad3-4249df44b31f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:39.854624 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.854600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6102ddf-5bed-4f4c-bad3-4249df44b31f" (UID: "d6102ddf-5bed-4f4c-bad3-4249df44b31f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:01:39.854708 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.854625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-dshm" (OuterVolumeSpecName: "dshm") pod "d6102ddf-5bed-4f4c-bad3-4249df44b31f" (UID: "d6102ddf-5bed-4f4c-bad3-4249df44b31f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:39.854708 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.854691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6102ddf-5bed-4f4c-bad3-4249df44b31f-kube-api-access-bb8tm" (OuterVolumeSpecName: "kube-api-access-bb8tm") pod "d6102ddf-5bed-4f4c-bad3-4249df44b31f" (UID: "d6102ddf-5bed-4f4c-bad3-4249df44b31f"). InnerVolumeSpecName "kube-api-access-bb8tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:01:39.953854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.953821    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:39.953854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.953847    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6102ddf-5bed-4f4c-bad3-4249df44b31f-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:39.953854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.953856    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6102ddf-5bed-4f4c-bad3-4249df44b31f-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:39.954118 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.953866    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb8tm\" (UniqueName: \"kubernetes.io/projected/d6102ddf-5bed-4f4c-bad3-4249df44b31f-kube-api-access-bb8tm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:39.957939 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.957914    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6102ddf-5bed-4f4c-bad3-4249df44b31f" containerID="247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c" exitCode=0
Sep 11 00:01:39.958094 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.957954    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"
Sep 11 00:01:39.958094 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.958006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64" event={"ID":"d6102ddf-5bed-4f4c-bad3-4249df44b31f","Type":"ContainerDied","Data":"247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c"}
Sep 11 00:01:39.958094 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.958055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64" event={"ID":"d6102ddf-5bed-4f4c-bad3-4249df44b31f","Type":"ContainerDied","Data":"411e39736aee0c589bd37ae8bca62830696176e938d1d7b73ebcff5d6a6e246e"}
Sep 11 00:01:39.958094 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.958079    2578 scope.go:117] "RemoveContainer" containerID="247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c"
Sep 11 00:01:39.967870 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.967852    2578 scope.go:117] "RemoveContainer" containerID="247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c"
Sep 11 00:01:39.968139 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:01:39.968121    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c\": container with ID starting with 247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c not found: ID does not exist" containerID="247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c"
Sep 11 00:01:39.968190 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.968149    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c"} err="failed to get container status \"247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c\": rpc error: code = NotFound desc = could not find container \"247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c\": container with ID starting with 247b6961550edc8fa4004c0683f5f6e94469b81ad1e90f05a70a77fd78bf320c not found: ID does not exist"
Sep 11 00:01:39.980311 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.980268    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"]
Sep 11 00:01:39.981918 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:39.981898    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/sub-test-model-6fa41e-kserve-8574dc6569-d7z64"]
Sep 11 00:01:40.182948 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.182908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:01:40.333304 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333248    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"]
Sep 11 00:01:40.333852 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333829    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90f1b9b-6254-47c4-a44e-41ea43b10f62" containerName="payload-processing"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333856    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90f1b9b-6254-47c4-a44e-41ea43b10f62" containerName="payload-processing"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333873    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" containerName="payload-pre-processing"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333883    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" containerName="payload-pre-processing"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333894    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" containerName="payload-processing"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" containerName="payload-processing"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333917    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6102ddf-5bed-4f4c-bad3-4249df44b31f" containerName="main"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333926    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6102ddf-5bed-4f4c-bad3-4249df44b31f" containerName="main"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333940    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333948    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333965    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859f5875-2761-4211-b221-9a91ab9d770f" containerName="payload-processing"
Sep 11 00:01:40.333972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.333973    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="859f5875-2761-4211-b221-9a91ab9d770f" containerName="payload-processing"
Sep 11 00:01:40.334465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.334088    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="859f5875-2761-4211-b221-9a91ab9d770f" containerName="payload-processing"
Sep 11 00:01:40.334465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.334104    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90f1b9b-6254-47c4-a44e-41ea43b10f62" containerName="payload-processing"
Sep 11 00:01:40.334465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.334114    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy"
Sep 11 00:01:40.334465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.334124    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc0ca52-68c9-41f1-b250-cbe0ec92e57d" containerName="payload-pre-processing"
Sep 11 00:01:40.334465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.334134    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e1ced1-d335-4f57-89bf-9ce7ed1dd4ed" containerName="payload-processing"
Sep 11 00:01:40.334465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.334141    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6102ddf-5bed-4f4c-bad3-4249df44b31f" containerName="main"
Sep 11 00:01:40.334465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.334150    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy"
Sep 11 00:01:40.339076 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.339051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.342097 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.341877    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"openshift-service-ca.crt\""
Sep 11 00:01:40.342097 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.341911    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"rate-test-model-6fa41e-kserve-self-signed-certs\""
Sep 11 00:01:40.342097 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.341947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"default-dockercfg-rnn8j\""
Sep 11 00:01:40.342667 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.342294    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6fa41e\"/\"kube-root-ca.crt\""
Sep 11 00:01:40.345707 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.345677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"]
Sep 11 00:01:40.458485 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.458399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b54a94-e4bf-4183-85a5-4785dd5de899-tls-certs\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.458485 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.458443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-tmp-dir\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.458485 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.458473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-dshm\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.458712 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.458518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hpkw\" (UniqueName: \"kubernetes.io/projected/45b54a94-e4bf-4183-85a5-4785dd5de899-kube-api-access-7hpkw\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.458712 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.458547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-model-cache\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.458712 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.458576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-home\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.504517 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.504484    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:40.537711 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.537671    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6102ddf-5bed-4f4c-bad3-4249df44b31f" path="/var/lib/kubelet/pods/d6102ddf-5bed-4f4c-bad3-4249df44b31f/volumes"
Sep 11 00:01:40.559228 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b54a94-e4bf-4183-85a5-4785dd5de899-tls-certs\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559413 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-tmp-dir\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559413 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-dshm\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559413 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpkw\" (UniqueName: \"kubernetes.io/projected/45b54a94-e4bf-4183-85a5-4785dd5de899-kube-api-access-7hpkw\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559413 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-model-cache\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559413 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-home\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559683 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-tmp-dir\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559752 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-home\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.559797 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.559753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-model-cache\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.561627 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.561604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-dshm\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.561936 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.561911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b54a94-e4bf-4183-85a5-4785dd5de899-tls-certs\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.571614 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.571594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpkw\" (UniqueName: \"kubernetes.io/projected/45b54a94-e4bf-4183-85a5-4785dd5de899-kube-api-access-7hpkw\") pod \"rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") " pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.637965 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.637922    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck" podUID="f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" containerName="main" containerID="cri-o://fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e" gracePeriod=45
Sep 11 00:01:40.651727 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.651702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:40.868304 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.868258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:01:40.920495 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.920471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"]
Sep 11 00:01:40.922241 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:01:40.922218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b54a94_e4bf_4183_85a5_4785dd5de899.slice/crio-e87abf7f61ab85f4d62e76a0f0e0ecd492e3b4ef3ee5e7d8a30b07b1fe552dbc WatchSource:0}: Error finding container e87abf7f61ab85f4d62e76a0f0e0ecd492e3b4ef3ee5e7d8a30b07b1fe552dbc: Status 404 returned error can't find the container with id e87abf7f61ab85f4d62e76a0f0e0ecd492e3b4ef3ee5e7d8a30b07b1fe552dbc
Sep 11 00:01:40.963329 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-home\") pod \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") "
Sep 11 00:01:40.963450 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-dshm\") pod \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") "
Sep 11 00:01:40.963450 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963412    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tls-certs\") pod \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") "
Sep 11 00:01:40.963578 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tmp-dir\") pod \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") "
Sep 11 00:01:40.963578 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f46gk\" (UniqueName: \"kubernetes.io/projected/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-kube-api-access-f46gk\") pod \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") "
Sep 11 00:01:40.963578 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963536    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-model-cache\") pod \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\" (UID: \"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744\") "
Sep 11 00:01:40.963715 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-home" (OuterVolumeSpecName: "home") pod "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" (UID: "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:40.963889 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.963866    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:40.964089 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.964015    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" (UID: "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:40.964412 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.964387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" (UID: "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:40.966172 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.966081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" (UID: "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:01:40.966172 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.966110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-dshm" (OuterVolumeSpecName: "dshm") pod "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" (UID: "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:01:40.966361 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.966171    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-kube-api-access-f46gk" (OuterVolumeSpecName: "kube-api-access-f46gk") pod "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" (UID: "f3161a16-1d6e-4c82-a2c6-7f52a2f3d744"). InnerVolumeSpecName "kube-api-access-f46gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:01:40.966554 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.966527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv" event={"ID":"45b54a94-e4bf-4183-85a5-4785dd5de899","Type":"ContainerStarted","Data":"e87abf7f61ab85f4d62e76a0f0e0ecd492e3b4ef3ee5e7d8a30b07b1fe552dbc"}
Sep 11 00:01:40.970709 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.968631    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"
Sep 11 00:01:40.970889 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.970858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck" event={"ID":"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744","Type":"ContainerDied","Data":"fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e"}
Sep 11 00:01:40.971005 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.970858    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" containerID="fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e" exitCode=0
Sep 11 00:01:40.971005 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.970918    2578 scope.go:117] "RemoveContainer" containerID="fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e"
Sep 11 00:01:40.971124 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.971016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck" event={"ID":"f3161a16-1d6e-4c82-a2c6-7f52a2f3d744","Type":"ContainerDied","Data":"2f494900ad2531db4bb5da07d6df4a1c5f10da54b2c95e8811bd8d2a6d4d4d3e"}
Sep 11 00:01:40.979485 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.979449    2578 scope.go:117] "RemoveContainer" containerID="fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e"
Sep 11 00:01:40.979731 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:01:40.979713    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e\": container with ID starting with fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e not found: ID does not exist" containerID="fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e"
Sep 11 00:01:40.979810 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.979738    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e"} err="failed to get container status \"fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e\": rpc error: code = NotFound desc = could not find container \"fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e\": container with ID starting with fecf3eb0148eaa767946ed14580529674c0b6105e11365b63edb610afd45dc9e not found: ID does not exist"
Sep 11 00:01:40.992586 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.992562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"]
Sep 11 00:01:40.998443 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:40.998420    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/sub-test-model-37c598-kserve-688bdc94cc-nztck"]
Sep 11 00:01:41.065021 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:41.064981    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:41.065021 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:41.065014    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46gk\" (UniqueName: \"kubernetes.io/projected/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-kube-api-access-f46gk\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:41.065021 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:41.065027    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:41.065267 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:41.065042    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:41.065267 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:41.065052    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:01:41.978028 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:41.977933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv" event={"ID":"45b54a94-e4bf-4183-85a5-4785dd5de899","Type":"ContainerStarted","Data":"f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7"}
Sep 11 00:01:41.994995 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:41.994920    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv" podStartSLOduration=1.994904196 podStartE2EDuration="1.994904196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:41.994190166 +0000 UTC m=+1543.950421970" watchObservedRunningTime="2026-09-11 00:01:41.994904196 +0000 UTC m=+1543.951135999"
Sep 11 00:01:42.535310 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:42.535255    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" path="/var/lib/kubelet/pods/f3161a16-1d6e-4c82-a2c6-7f52a2f3d744/volumes"
Sep 11 00:01:43.945029 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:43.944994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:45.431830 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:45.431786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:47.820737 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:47.820705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:48.487288 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:48.487244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:49.906053 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:49.906015    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:01:50.652063 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:50.652027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:50.652461 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:50.652432    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:50.670588 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:50.670560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:51.037421 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:51.037386    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:01:52.286501 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:52.286468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:52.347803 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:52.347769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:52.556778 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:01:52.556691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:01.151008 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:01.150975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:05.542488 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:05.542456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:11.108925 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:11.108879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:11.211103 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:11.211062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:15.384390 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:15.384343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:24.440381 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.440343    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"]
Sep 11 00:02:24.440896 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.440736    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy"
Sep 11 00:02:24.440896 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.440751    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="334c6b48-0d1d-414e-9117-4c401741c484" containerName="istio-proxy"
Sep 11 00:02:24.440896 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.440763    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" containerName="main"
Sep 11 00:02:24.440896 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.440773    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" containerName="main"
Sep 11 00:02:24.440896 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.440845    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3161a16-1d6e-4c82-a2c6-7f52a2f3d744" containerName="main"
Sep 11 00:02:24.444123 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.444099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.448106 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.448080    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"default-dockercfg-rxxx8\""
Sep 11 00:02:24.448254 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.448080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"openshift-service-ca.crt\""
Sep 11 00:02:24.448254 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.448083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"kube-root-ca.crt\""
Sep 11 00:02:24.448254 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.448083    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-37c598\"/\"rate-test-model-37c598-kserve-self-signed-certs\""
Sep 11 00:02:24.454789 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.454765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"]
Sep 11 00:02:24.464772 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.464735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:24.480685 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.480651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-model-cache\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.480832 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.480698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tmp-dir\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.480832 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.480756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-dshm\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.480832 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.480784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2fb\" (UniqueName: \"kubernetes.io/projected/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-kube-api-access-9c2fb\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.480832 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.480827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-home\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.481075 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.480843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tls-certs\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.581966 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.581931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2fb\" (UniqueName: \"kubernetes.io/projected/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-kube-api-access-9c2fb\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.581966 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.581980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-home\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.582240 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.581998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tls-certs\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.582240 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.582117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-model-cache\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.582240 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.582190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tmp-dir\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.582459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.582262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-dshm\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.582459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.582432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-home\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.582538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.582477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-model-cache\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.582538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.582513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tmp-dir\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.584333 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.584312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-dshm\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.584538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.584521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tls-certs\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.592433 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.592407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2fb\" (UniqueName: \"kubernetes.io/projected/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-kube-api-access-9c2fb\") pod \"rate-test-model-37c598-kserve-78b7bdf74c-s9pc4\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") " pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:24.755024 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:24.754922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:25.024043 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:25.024015    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"]
Sep 11 00:02:25.025883 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:02:25.025854    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c00a16a_4cbb_44b7_9ac6_04883c6f5623.slice/crio-77373aa5695354651cea94816bdc6b1c574097b9fa000c92030443e7a534abda WatchSource:0}: Error finding container 77373aa5695354651cea94816bdc6b1c574097b9fa000c92030443e7a534abda: Status 404 returned error can't find the container with id 77373aa5695354651cea94816bdc6b1c574097b9fa000c92030443e7a534abda
Sep 11 00:02:25.193156 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:25.193119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4" event={"ID":"4c00a16a-4cbb-44b7-9ac6-04883c6f5623","Type":"ContainerStarted","Data":"ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69"}
Sep 11 00:02:25.193355 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:25.193164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4" event={"ID":"4c00a16a-4cbb-44b7-9ac6-04883c6f5623","Type":"ContainerStarted","Data":"77373aa5695354651cea94816bdc6b1c574097b9fa000c92030443e7a534abda"}
Sep 11 00:02:25.211972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:25.211926    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4" podStartSLOduration=1.211909671 podStartE2EDuration="1.211909671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:02:25.210324352 +0000 UTC m=+1587.166556155" watchObservedRunningTime="2026-09-11 00:02:25.211909671 +0000 UTC m=+1587.168141476"
Sep 11 00:02:27.944161 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:27.944126    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:02:34.755045 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:34.755011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:34.755515 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:34.755070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:34.767588 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:34.767564    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:35.251291 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:35.251245    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:02:53.698078 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:02:53.698001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:03:09.189062 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:09.189029    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:03:10.377016 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:10.376958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:13.823891 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:13.823852    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"]
Sep 11 00:03:13.839136 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:13.839088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:14.967359 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:14.967324    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"]
Sep 11 00:03:14.987068 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:14.987041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:15.394898 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:15.394849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:15.691221 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:15.691139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:16.289602 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:16.289563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:16.483442 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:16.483409    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:03:18.365454 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:18.365418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:19.141515 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:19.141478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:19.236854 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:19.236812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:20.099807 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:20.099763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:20.894852 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:20.894817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:20.966263 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:20.966205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:21.066525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.066491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:21.390563 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.390533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:21.475505 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.475461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:21.521383 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.521347    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"]
Sep 11 00:03:21.525011 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.524990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:21.527637 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.527619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7c3755-dockercfg-pv6jq\""
Sep 11 00:03:21.527745 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.527713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7c3755\""
Sep 11 00:03:21.536208 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.536188    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"]
Sep 11 00:03:21.613023 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.612987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:21.613212 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.613043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtk2\" (UniqueName: \"kubernetes.io/projected/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-kube-api-access-wbtk2\") pod \"payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:21.713874 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.713789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:21.713874 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.713838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtk2\" (UniqueName: \"kubernetes.io/projected/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-kube-api-access-wbtk2\") pod \"payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:21.714550 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.714526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:21.721445 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.721420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtk2\" (UniqueName: \"kubernetes.io/projected/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-kube-api-access-wbtk2\") pod \"payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:21.836004 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:21.835970    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:22.106230 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:22.106201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"]
Sep 11 00:03:22.108073 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:03:22.108039    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd7a5eb_84df_4f3a_9efd_f6f8e5caa5c0.slice/crio-a86a19cacb93d640403c8e0de2beeaef9231a217b7f23cf96742e46fac4f2e36 WatchSource:0}: Error finding container a86a19cacb93d640403c8e0de2beeaef9231a217b7f23cf96742e46fac4f2e36: Status 404 returned error can't find the container with id a86a19cacb93d640403c8e0de2beeaef9231a217b7f23cf96742e46fac4f2e36
Sep 11 00:03:22.475679 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:22.475585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs" event={"ID":"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0","Type":"ContainerStarted","Data":"652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6"}
Sep 11 00:03:22.475679 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:22.475636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs" event={"ID":"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0","Type":"ContainerStarted","Data":"a86a19cacb93d640403c8e0de2beeaef9231a217b7f23cf96742e46fac4f2e36"}
Sep 11 00:03:22.476093 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:22.475680    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:22.493961 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:22.493911    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs" podStartSLOduration=1.493898287 podStartE2EDuration="1.493898287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:22.493253203 +0000 UTC m=+1644.449485007" watchObservedRunningTime="2026-09-11 00:03:22.493898287 +0000 UTC m=+1644.450130088"
Sep 11 00:03:24.590796 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:24.590762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:03:28.844084 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:28.844038    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv" podUID="45b54a94-e4bf-4183-85a5-4785dd5de899" containerName="main" containerID="cri-o://f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7" gracePeriod=45
Sep 11 00:03:29.089187 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.089158    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:03:29.186671 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-model-cache\") pod \"45b54a94-e4bf-4183-85a5-4785dd5de899\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") "
Sep 11 00:03:29.186671 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-home\") pod \"45b54a94-e4bf-4183-85a5-4785dd5de899\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") "
Sep 11 00:03:29.186876 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hpkw\" (UniqueName: \"kubernetes.io/projected/45b54a94-e4bf-4183-85a5-4785dd5de899-kube-api-access-7hpkw\") pod \"45b54a94-e4bf-4183-85a5-4785dd5de899\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") "
Sep 11 00:03:29.186876 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-dshm\") pod \"45b54a94-e4bf-4183-85a5-4785dd5de899\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") "
Sep 11 00:03:29.186952 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-model-cache" (OuterVolumeSpecName: "model-cache") pod "45b54a94-e4bf-4183-85a5-4785dd5de899" (UID: "45b54a94-e4bf-4183-85a5-4785dd5de899"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:29.186952 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b54a94-e4bf-4183-85a5-4785dd5de899-tls-certs\") pod \"45b54a94-e4bf-4183-85a5-4785dd5de899\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") "
Sep 11 00:03:29.186952 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186938    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-home" (OuterVolumeSpecName: "home") pod "45b54a94-e4bf-4183-85a5-4785dd5de899" (UID: "45b54a94-e4bf-4183-85a5-4785dd5de899"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:29.187069 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.186947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-tmp-dir\") pod \"45b54a94-e4bf-4183-85a5-4785dd5de899\" (UID: \"45b54a94-e4bf-4183-85a5-4785dd5de899\") "
Sep 11 00:03:29.187226 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.187198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45b54a94-e4bf-4183-85a5-4785dd5de899" (UID: "45b54a94-e4bf-4183-85a5-4785dd5de899"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:29.187363 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.187347    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:29.187438 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.187364    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:29.187438 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.187378    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:29.188913 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.188886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b54a94-e4bf-4183-85a5-4785dd5de899-kube-api-access-7hpkw" (OuterVolumeSpecName: "kube-api-access-7hpkw") pod "45b54a94-e4bf-4183-85a5-4785dd5de899" (UID: "45b54a94-e4bf-4183-85a5-4785dd5de899"). InnerVolumeSpecName "kube-api-access-7hpkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:03:29.189289 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.189256    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b54a94-e4bf-4183-85a5-4785dd5de899-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45b54a94-e4bf-4183-85a5-4785dd5de899" (UID: "45b54a94-e4bf-4183-85a5-4785dd5de899"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:03:29.189364 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.189256    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-dshm" (OuterVolumeSpecName: "dshm") pod "45b54a94-e4bf-4183-85a5-4785dd5de899" (UID: "45b54a94-e4bf-4183-85a5-4785dd5de899"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:29.288132 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.288099    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hpkw\" (UniqueName: \"kubernetes.io/projected/45b54a94-e4bf-4183-85a5-4785dd5de899-kube-api-access-7hpkw\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:29.288132 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.288129    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b54a94-e4bf-4183-85a5-4785dd5de899-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:29.288348 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.288141    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b54a94-e4bf-4183-85a5-4785dd5de899-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:29.509847 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.509760    2578 generic.go:356] "Generic (PLEG): container finished" podID="45b54a94-e4bf-4183-85a5-4785dd5de899" containerID="f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7" exitCode=0
Sep 11 00:03:29.509847 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.509801    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"
Sep 11 00:03:29.510038 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.509842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv" event={"ID":"45b54a94-e4bf-4183-85a5-4785dd5de899","Type":"ContainerDied","Data":"f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7"}
Sep 11 00:03:29.510038 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.509883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv" event={"ID":"45b54a94-e4bf-4183-85a5-4785dd5de899","Type":"ContainerDied","Data":"e87abf7f61ab85f4d62e76a0f0e0ecd492e3b4ef3ee5e7d8a30b07b1fe552dbc"}
Sep 11 00:03:29.510038 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.509900    2578 scope.go:117] "RemoveContainer" containerID="f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7"
Sep 11 00:03:29.519442 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.519420    2578 scope.go:117] "RemoveContainer" containerID="f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7"
Sep 11 00:03:29.519723 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:03:29.519706    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7\": container with ID starting with f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7 not found: ID does not exist" containerID="f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7"
Sep 11 00:03:29.519788 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.519733    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7"} err="failed to get container status \"f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7\": rpc error: code = NotFound desc = could not find container \"f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7\": container with ID starting with f76901ee1424fa3589be1c133b2f3ddeb88e9e9b8e15a37b6fd87d68ce1e48c7 not found: ID does not exist"
Sep 11 00:03:29.531454 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.531431    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"]
Sep 11 00:03:29.534972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.534951    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6fa41e/rate-test-model-6fa41e-kserve-54cbbb584c-mhrtv"]
Sep 11 00:03:29.985710 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:29.985657    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4" podUID="4c00a16a-4cbb-44b7-9ac6-04883c6f5623" containerName="main" containerID="cri-o://ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69" gracePeriod=45
Sep 11 00:03:30.230500 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.230474    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:03:30.397792 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.397763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tmp-dir\") pod \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") "
Sep 11 00:03:30.397975 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.397808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c2fb\" (UniqueName: \"kubernetes.io/projected/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-kube-api-access-9c2fb\") pod \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") "
Sep 11 00:03:30.397975 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.397832    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-model-cache\") pod \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") "
Sep 11 00:03:30.397975 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.397850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tls-certs\") pod \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") "
Sep 11 00:03:30.397975 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.397905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-home\") pod \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") "
Sep 11 00:03:30.398197 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.397997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-dshm\") pod \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\" (UID: \"4c00a16a-4cbb-44b7-9ac6-04883c6f5623\") "
Sep 11 00:03:30.398197 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.398080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c00a16a-4cbb-44b7-9ac6-04883c6f5623" (UID: "4c00a16a-4cbb-44b7-9ac6-04883c6f5623"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:30.398197 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.398100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c00a16a-4cbb-44b7-9ac6-04883c6f5623" (UID: "4c00a16a-4cbb-44b7-9ac6-04883c6f5623"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:30.398360 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.398235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-home" (OuterVolumeSpecName: "home") pod "4c00a16a-4cbb-44b7-9ac6-04883c6f5623" (UID: "4c00a16a-4cbb-44b7-9ac6-04883c6f5623"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:30.398400 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.398370    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:30.398400 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.398384    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:30.398400 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.398392    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:30.400056 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.400027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c00a16a-4cbb-44b7-9ac6-04883c6f5623" (UID: "4c00a16a-4cbb-44b7-9ac6-04883c6f5623"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:03:30.400166 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.400069    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-dshm" (OuterVolumeSpecName: "dshm") pod "4c00a16a-4cbb-44b7-9ac6-04883c6f5623" (UID: "4c00a16a-4cbb-44b7-9ac6-04883c6f5623"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:03:30.400166 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.400099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-kube-api-access-9c2fb" (OuterVolumeSpecName: "kube-api-access-9c2fb") pod "4c00a16a-4cbb-44b7-9ac6-04883c6f5623" (UID: "4c00a16a-4cbb-44b7-9ac6-04883c6f5623"). InnerVolumeSpecName "kube-api-access-9c2fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:03:30.499218 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.499182    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:30.499218 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.499213    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2fb\" (UniqueName: \"kubernetes.io/projected/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-kube-api-access-9c2fb\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:30.499423 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.499226    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c00a16a-4cbb-44b7-9ac6-04883c6f5623-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:03:30.516510 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.516484    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c00a16a-4cbb-44b7-9ac6-04883c6f5623" containerID="ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69" exitCode=0
Sep 11 00:03:30.516664 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.516525    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"
Sep 11 00:03:30.516664 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.516577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4" event={"ID":"4c00a16a-4cbb-44b7-9ac6-04883c6f5623","Type":"ContainerDied","Data":"ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69"}
Sep 11 00:03:30.516664 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.516628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4" event={"ID":"4c00a16a-4cbb-44b7-9ac6-04883c6f5623","Type":"ContainerDied","Data":"77373aa5695354651cea94816bdc6b1c574097b9fa000c92030443e7a534abda"}
Sep 11 00:03:30.516664 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.516657    2578 scope.go:117] "RemoveContainer" containerID="ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69"
Sep 11 00:03:30.525189 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.525170    2578 scope.go:117] "RemoveContainer" containerID="ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69"
Sep 11 00:03:30.525489 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:03:30.525471    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69\": container with ID starting with ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69 not found: ID does not exist" containerID="ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69"
Sep 11 00:03:30.525556 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.525497    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69"} err="failed to get container status \"ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69\": rpc error: code = NotFound desc = could not find container \"ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69\": container with ID starting with ff5fcdaa220adef17be4058df55221301ccc8ef5070ac02feeba2b7e8ce2ae69 not found: ID does not exist"
Sep 11 00:03:30.534213 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.534186    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b54a94-e4bf-4183-85a5-4785dd5de899" path="/var/lib/kubelet/pods/45b54a94-e4bf-4183-85a5-4785dd5de899/volumes"
Sep 11 00:03:30.537145 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.537123    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"]
Sep 11 00:03:30.542769 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:30.542747    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-37c598/rate-test-model-37c598-kserve-78b7bdf74c-s9pc4"]
Sep 11 00:03:32.533726 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:32.533692    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c00a16a-4cbb-44b7-9ac6-04883c6f5623" path="/var/lib/kubelet/pods/4c00a16a-4cbb-44b7-9ac6-04883c6f5623/volumes"
Sep 11 00:03:33.483455 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:33.483423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:03:43.081218 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:43.081167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:43.812523 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:43.812483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:44.078732 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:44.078640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:44.557568 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:44.557530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:44.757754 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:44.757715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:45.049324 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.049285    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"]
Sep 11 00:03:45.049847 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.049832    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b54a94-e4bf-4183-85a5-4785dd5de899" containerName="main"
Sep 11 00:03:45.049903 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.049851    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b54a94-e4bf-4183-85a5-4785dd5de899" containerName="main"
Sep 11 00:03:45.049903 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.049866    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c00a16a-4cbb-44b7-9ac6-04883c6f5623" containerName="main"
Sep 11 00:03:45.049903 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.049875    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c00a16a-4cbb-44b7-9ac6-04883c6f5623" containerName="main"
Sep 11 00:03:45.050006 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.049988    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b54a94-e4bf-4183-85a5-4785dd5de899" containerName="main"
Sep 11 00:03:45.050006 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.050003    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c00a16a-4cbb-44b7-9ac6-04883c6f5623" containerName="main"
Sep 11 00:03:45.054859 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.054834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.057532 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.057511    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-da6814-openshift-default-dockercfg-mmzzq\""
Sep 11 00:03:45.063436 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.063412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"]
Sep 11 00:03:45.081369 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.081333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:45.131677 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-socket\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.131848 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-data\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.131848 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-certs\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.131848 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8hd\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-kube-api-access-kk8hd\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.131848 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-envoy\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.131848 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-podinfo\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.131848 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220c1333-bb6f-47bf-ac86-883cba46d4fc-istiod-ca-cert\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.132089 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-credential-socket\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.132089 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.131912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-token\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.232838 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-socket\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-data\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-certs\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8hd\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-kube-api-access-kk8hd\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-envoy\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-podinfo\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220c1333-bb6f-47bf-ac86-883cba46d4fc-istiod-ca-cert\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.232986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-credential-socket\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233410 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.233027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-token\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233410 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.233264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-socket\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233410 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.233297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-data\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233528 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.233411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-certs\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233570 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.233533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-credential-socket\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.233823 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.233797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220c1333-bb6f-47bf-ac86-883cba46d4fc-istiod-ca-cert\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.235317 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.235265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-envoy\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.235452 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.235428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-podinfo\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.243489 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.243452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-token\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.243768 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.243750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8hd\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-kube-api-access-kk8hd\") pod \"e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") " pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.366400 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.366299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:45.439793 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.439697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:45.539519 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.539480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:45.642656 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.642632    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"]
Sep 11 00:03:45.645156 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:03:45.645119    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220c1333_bb6f_47bf_ac86_883cba46d4fc.slice/crio-2ab0833856117a2de68c8286ce89cca17e01cca0fb26c818e13fd5bf1b31d6b8 WatchSource:0}: Error finding container 2ab0833856117a2de68c8286ce89cca17e01cca0fb26c818e13fd5bf1b31d6b8: Status 404 returned error can't find the container with id 2ab0833856117a2de68c8286ce89cca17e01cca0fb26c818e13fd5bf1b31d6b8
Sep 11 00:03:45.647818 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.647786    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:45.647895 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.647855    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:45.647895 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:45.647888    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:46.597571 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:46.597535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" event={"ID":"220c1333-bb6f-47bf-ac86-883cba46d4fc","Type":"ContainerStarted","Data":"4820779e0296a3aeeae592f552550c6d6b7e5ec2ec4b732933ed4071987a7664"}
Sep 11 00:03:46.597754 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:46.597599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" event={"ID":"220c1333-bb6f-47bf-ac86-883cba46d4fc","Type":"ContainerStarted","Data":"2ab0833856117a2de68c8286ce89cca17e01cca0fb26c818e13fd5bf1b31d6b8"}
Sep 11 00:03:46.618597 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:46.618547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" podStartSLOduration=1.618533277 podStartE2EDuration="1.618533277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:46.616042627 +0000 UTC m=+1668.572274438" watchObservedRunningTime="2026-09-11 00:03:46.618533277 +0000 UTC m=+1668.574765080"
Sep 11 00:03:47.367127 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:47.367090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:47.372526 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:47.372499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:47.603211 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:47.603181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:47.604175 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:47.604156    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:03:48.077491 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:48.077454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:48.490835 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:48.490751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:48.628561 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:48.628523    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:48.867234 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:48.867199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:49.765129 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:49.765097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:50.468114 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:50.468073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:51.284894 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.284863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:51.385691 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.385655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:51.771870 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.771837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:51.856980 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.856942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:51.887561 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.887521    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"]
Sep 11 00:03:51.891561 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.891541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:51.894188 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.894158    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-da6814\""
Sep 11 00:03:51.894333 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.894318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-da6814-dockercfg-dlq6x\""
Sep 11 00:03:51.900500 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:51.900475    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"]
Sep 11 00:03:52.000650 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.000609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgg4l\" (UniqueName: \"kubernetes.io/projected/0558a194-b0ed-4998-b880-dc27fe123758-kube-api-access-sgg4l\") pod \"payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.000822 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.000662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0558a194-b0ed-4998-b880-dc27fe123758-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.101828 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.101740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgg4l\" (UniqueName: \"kubernetes.io/projected/0558a194-b0ed-4998-b880-dc27fe123758-kube-api-access-sgg4l\") pod \"payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.101828 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.101806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0558a194-b0ed-4998-b880-dc27fe123758-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.102542 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.102522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0558a194-b0ed-4998-b880-dc27fe123758-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.109482 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.109452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgg4l\" (UniqueName: \"kubernetes.io/projected/0558a194-b0ed-4998-b880-dc27fe123758-kube-api-access-sgg4l\") pod \"payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.204232 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.204196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.476373 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.476343    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"]
Sep 11 00:03:52.478528 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:03:52.478502    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0558a194_b0ed_4998_b880_dc27fe123758.slice/crio-7886d4eb0c0684756c840135a52c86afde6947ddcad7570bbb19e3a2d1cfa60d WatchSource:0}: Error finding container 7886d4eb0c0684756c840135a52c86afde6947ddcad7570bbb19e3a2d1cfa60d: Status 404 returned error can't find the container with id 7886d4eb0c0684756c840135a52c86afde6947ddcad7570bbb19e3a2d1cfa60d
Sep 11 00:03:52.629412 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.629377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j" event={"ID":"0558a194-b0ed-4998-b880-dc27fe123758","Type":"ContainerStarted","Data":"1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff"}
Sep 11 00:03:52.629573 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.629429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j" event={"ID":"0558a194-b0ed-4998-b880-dc27fe123758","Type":"ContainerStarted","Data":"7886d4eb0c0684756c840135a52c86afde6947ddcad7570bbb19e3a2d1cfa60d"}
Sep 11 00:03:52.629573 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.629477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:03:52.646029 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:52.645984    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j" podStartSLOduration=1.645970454 podStartE2EDuration="1.645970454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:52.644453913 +0000 UTC m=+1674.600685717" watchObservedRunningTime="2026-09-11 00:03:52.645970454 +0000 UTC m=+1674.602202256"
Sep 11 00:03:55.894132 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:03:55.894099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:04:03.639731 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:03.639689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:04:06.709942 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:06.709909    2578 generic.go:356] "Generic (PLEG): container finished" podID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerID="4820779e0296a3aeeae592f552550c6d6b7e5ec2ec4b732933ed4071987a7664" exitCode=0
Sep 11 00:04:06.710397 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:06.709975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" event={"ID":"220c1333-bb6f-47bf-ac86-883cba46d4fc","Type":"ContainerDied","Data":"4820779e0296a3aeeae592f552550c6d6b7e5ec2ec4b732933ed4071987a7664"}
Sep 11 00:04:06.710397 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:06.710378    2578 scope.go:117] "RemoveContainer" containerID="4820779e0296a3aeeae592f552550c6d6b7e5ec2ec4b732933ed4071987a7664"
Sep 11 00:04:06.711564 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:06.711534    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:04:06.711639 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:06.711605    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:04:06.711676 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:06.711636    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:04:07.723009 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:07.722972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" event={"ID":"220c1333-bb6f-47bf-ac86-883cba46d4fc","Type":"ContainerStarted","Data":"b3081405ac09921f312f1f2e4353f6526e8a6f3b5da5838651eef6d1cc00267a"}
Sep 11 00:04:08.366700 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:08.366656    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:04:08.367002 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:08.366971    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 00:04:08.367149 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:08.367023    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 00:04:09.366934 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:09.366894    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 00:04:09.367383 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:09.366952    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 00:04:10.366677 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:10.366639    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 00:04:10.366847 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:10.366695    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 00:04:11.367128 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:11.367087    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 00:04:11.367553 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:11.367144    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 00:04:12.366691 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:12.366649    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 00:04:12.366906 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:12.366756    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 00:04:13.371003 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:13.370970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:04:13.371443 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:13.371239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:04:13.372098 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:13.372073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:04:13.376754 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:13.376732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:13.486210 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:13.486171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:14.306340 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:14.306308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:15.048591 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:15.048558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:15.185731 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:15.185692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:15.974090 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:15.974059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:16.659411 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:16.659343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:18.555493 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:18.555461    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j_0558a194-b0ed-4998-b880-dc27fe123758/payload-processing/0.log"
Sep 11 00:04:27.733817 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.733741    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"]
Sep 11 00:04:27.738435 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.738415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.742934 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.742768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7c3755\"/\"default-dockercfg-ddcbb\""
Sep 11 00:04:27.742934 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.742808    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7c3755\"/\"openshift-service-ca.crt\""
Sep 11 00:04:27.743151 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.742974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7c3755\"/\"kube-root-ca.crt\""
Sep 11 00:04:27.743151 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.743069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7c3755\"/\"ipp-route-7c3755-kserve-self-signed-certs\""
Sep 11 00:04:27.745968 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.745947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"]
Sep 11 00:04:27.844454 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.844424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tmp-dir\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.844633 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.844469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-model-cache\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.844633 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.844491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tls-certs\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.844633 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.844531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-dshm\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.844633 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.844547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64f48\" (UniqueName: \"kubernetes.io/projected/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-kube-api-access-64f48\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.844633 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.844578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-home\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.919368 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.919338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:27.945351 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tmp-dir\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-model-cache\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tls-certs\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-dshm\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64f48\" (UniqueName: \"kubernetes.io/projected/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-kube-api-access-64f48\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-home\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945807 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tmp-dir\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945875 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-model-cache\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.945930 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.945873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-home\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.947661 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.947629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-dshm\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.947924 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.947908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tls-certs\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:27.963715 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:27.963690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64f48\" (UniqueName: \"kubernetes.io/projected/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-kube-api-access-64f48\") pod \"ipp-route-7c3755-kserve-64dbfc4995-cbrnz\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") " pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:28.048638 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:28.048597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:28.317624 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:28.317592    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"]
Sep 11 00:04:28.319318 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:04:28.319267    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c266ab1_f682_4eeb_8c9a_9b059297f1ae.slice/crio-55a7a6dc785f2caae60e122665458cb8f86622dc45f799ae84f5f696bb8d3ca0 WatchSource:0}: Error finding container 55a7a6dc785f2caae60e122665458cb8f86622dc45f799ae84f5f696bb8d3ca0: Status 404 returned error can't find the container with id 55a7a6dc785f2caae60e122665458cb8f86622dc45f799ae84f5f696bb8d3ca0
Sep 11 00:04:28.849975 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:28.849941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz" event={"ID":"3c266ab1-f682-4eeb-8c9a-9b059297f1ae","Type":"ContainerStarted","Data":"cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6"}
Sep 11 00:04:28.850379 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:28.849985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz" event={"ID":"3c266ab1-f682-4eeb-8c9a-9b059297f1ae","Type":"ContainerStarted","Data":"55a7a6dc785f2caae60e122665458cb8f86622dc45f799ae84f5f696bb8d3ca0"}
Sep 11 00:04:28.865842 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:28.865795    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz" podStartSLOduration=1.865781087 podStartE2EDuration="1.865781087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:04:28.864766561 +0000 UTC m=+1710.820998363" watchObservedRunningTime="2026-09-11 00:04:28.865781087 +0000 UTC m=+1710.822012953"
Sep 11 00:04:31.787161 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:31.787125    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:04:38.048970 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:38.048919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:38.048970 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:38.048972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:38.062257 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:38.062231    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:38.912095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:38.912065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:04:52.696149 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:52.696110    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:04:59.175999 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:59.175966    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs_7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0/payload-processing/0.log"
Sep 11 00:04:59.418412 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:59.418382    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j_0558a194-b0ed-4998-b880-dc27fe123758/payload-processing/0.log"
Sep 11 00:04:59.678118 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:59.678080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:59.989251 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:04:59.989159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:00.402223 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:00.402189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:03.029263 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:03.029212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:03.675814 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:03.675762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:03.776892 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:03.776855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:04.536162 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:04.536129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:05.416775 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:05.416745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:06.070367 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:06.070332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:06.172216 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:06.172184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:06.283171 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:06.283138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:06.992215 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:06.992187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:07.092733 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.092699    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:07.146268 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.146236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:07.462124 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.462095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:07.569305 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.569262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:07.610537 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.610504    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"]
Sep 11 00:05:07.614262 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.614242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:07.617332 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.617308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ef4657-dockercfg-rvn54\""
Sep 11 00:05:07.617436 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.617350    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ef4657\""
Sep 11 00:05:07.623504 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.623480    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"]
Sep 11 00:05:07.716382 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.716302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4hz\" (UniqueName: \"kubernetes.io/projected/49f1ecf8-943a-4833-af82-4fada16c7af5-kube-api-access-zw4hz\") pod \"payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:07.716382 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.716360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f1ecf8-943a-4833-af82-4fada16c7af5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:07.817324 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.817269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4hz\" (UniqueName: \"kubernetes.io/projected/49f1ecf8-943a-4833-af82-4fada16c7af5-kube-api-access-zw4hz\") pod \"payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:07.817500 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.817347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f1ecf8-943a-4833-af82-4fada16c7af5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:07.818016 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.817997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f1ecf8-943a-4833-af82-4fada16c7af5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:07.824962 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.824944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4hz\" (UniqueName: \"kubernetes.io/projected/49f1ecf8-943a-4833-af82-4fada16c7af5-kube-api-access-zw4hz\") pod \"payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:07.926291 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:07.926250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:08.199938 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:08.199903    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"]
Sep 11 00:05:08.201884 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:05:08.201852    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f1ecf8_943a_4833_af82_4fada16c7af5.slice/crio-3e7af7026c8d5846a77aca2bec5cfaba97f292819e487f8bbf4a3aaa776959e1 WatchSource:0}: Error finding container 3e7af7026c8d5846a77aca2bec5cfaba97f292819e487f8bbf4a3aaa776959e1: Status 404 returned error can't find the container with id 3e7af7026c8d5846a77aca2bec5cfaba97f292819e487f8bbf4a3aaa776959e1
Sep 11 00:05:09.052796 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:09.052760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77" event={"ID":"49f1ecf8-943a-4833-af82-4fada16c7af5","Type":"ContainerStarted","Data":"27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431"}
Sep 11 00:05:09.052971 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:09.052817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77" event={"ID":"49f1ecf8-943a-4833-af82-4fada16c7af5","Type":"ContainerStarted","Data":"3e7af7026c8d5846a77aca2bec5cfaba97f292819e487f8bbf4a3aaa776959e1"}
Sep 11 00:05:09.052971 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:09.052895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:09.071727 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:09.071682    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77" podStartSLOduration=2.071668795 podStartE2EDuration="2.071668795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:05:09.070746618 +0000 UTC m=+1751.026978422" watchObservedRunningTime="2026-09-11 00:05:09.071668795 +0000 UTC m=+1751.027900601"
Sep 11 00:05:11.692922 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:11.692881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:05:20.060836 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:20.060805    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:29.006892 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:29.006856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:29.883364 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:29.883322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:30.578850 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:30.578803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:31.264797 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:31.264760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:31.959905 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:31.959876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:32.655869 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:32.655823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:33.645390 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:33.645356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:35.192729 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:35.192681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:36.042026 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:36.041991    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:36.707041 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:36.707014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.617812 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:37.617779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:38.264298 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:38.264252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:39.095007 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:39.094974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:45.214513 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.214477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:45.244681 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.244647    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"]
Sep 11 00:05:45.244944 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.244894    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77" podUID="49f1ecf8-943a-4833-af82-4fada16c7af5" containerName="payload-processing" containerID="cri-o://27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431" gracePeriod=30
Sep 11 00:05:45.317470 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.317437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:45.415849 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.415740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:45.481122 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.481092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:45.504580 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.504553    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:45.564250 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.564221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f1ecf8-943a-4833-af82-4fada16c7af5-plugins-config-volume\") pod \"49f1ecf8-943a-4833-af82-4fada16c7af5\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") "
Sep 11 00:05:45.564423 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.564289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw4hz\" (UniqueName: \"kubernetes.io/projected/49f1ecf8-943a-4833-af82-4fada16c7af5-kube-api-access-zw4hz\") pod \"49f1ecf8-943a-4833-af82-4fada16c7af5\" (UID: \"49f1ecf8-943a-4833-af82-4fada16c7af5\") "
Sep 11 00:05:45.564594 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.564568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f1ecf8-943a-4833-af82-4fada16c7af5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49f1ecf8-943a-4833-af82-4fada16c7af5" (UID: "49f1ecf8-943a-4833-af82-4fada16c7af5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:45.566352 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.566327    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f1ecf8-943a-4833-af82-4fada16c7af5-kube-api-access-zw4hz" (OuterVolumeSpecName: "kube-api-access-zw4hz") pod "49f1ecf8-943a-4833-af82-4fada16c7af5" (UID: "49f1ecf8-943a-4833-af82-4fada16c7af5"). InnerVolumeSpecName "kube-api-access-zw4hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:05:45.665048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.665009    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f1ecf8-943a-4833-af82-4fada16c7af5-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:05:45.665048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.665045    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw4hz\" (UniqueName: \"kubernetes.io/projected/49f1ecf8-943a-4833-af82-4fada16c7af5-kube-api-access-zw4hz\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:05:45.778787 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.778755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:45.984068 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:45.984031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:46.254942 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.254851    2578 generic.go:356] "Generic (PLEG): container finished" podID="49f1ecf8-943a-4833-af82-4fada16c7af5" containerID="27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431" exitCode=0
Sep 11 00:05:46.254942 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.254899    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"
Sep 11 00:05:46.254942 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.254899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77" event={"ID":"49f1ecf8-943a-4833-af82-4fada16c7af5","Type":"ContainerDied","Data":"27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431"}
Sep 11 00:05:46.255494 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.254953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77" event={"ID":"49f1ecf8-943a-4833-af82-4fada16c7af5","Type":"ContainerDied","Data":"3e7af7026c8d5846a77aca2bec5cfaba97f292819e487f8bbf4a3aaa776959e1"}
Sep 11 00:05:46.255494 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.254983    2578 scope.go:117] "RemoveContainer" containerID="27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431"
Sep 11 00:05:46.264097 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.264078    2578 scope.go:117] "RemoveContainer" containerID="27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431"
Sep 11 00:05:46.264414 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:05:46.264393    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431\": container with ID starting with 27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431 not found: ID does not exist" containerID="27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431"
Sep 11 00:05:46.264461 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.264423    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431"} err="failed to get container status \"27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431\": rpc error: code = NotFound desc = could not find container \"27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431\": container with ID starting with 27ef3f948c70c3cbb58cc26af4d4d142cd3a7dbb418561b8258ac8956c3c0431 not found: ID does not exist"
Sep 11 00:05:46.277019 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.276996    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"]
Sep 11 00:05:46.280759 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.280737    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ef4657-94ff6fcd9-fvl77"]
Sep 11 00:05:46.540884 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.540848    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f1ecf8-943a-4833-af82-4fada16c7af5" path="/var/lib/kubelet/pods/49f1ecf8-943a-4833-af82-4fada16c7af5/volumes"
Sep 11 00:05:46.769763 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:46.769726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:47.690815 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:47.690766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:48.471382 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:48.471347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:49.195616 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:49.195579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:49.692861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:49.692828    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:05:52.075649 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:52.075560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:52.176438 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:52.176407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:52.346431 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:52.346353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:57.248434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:57.248400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:57.465630 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:57.465596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:58.165796 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:58.165762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:58.654687 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:58.654657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 11 00:05:58.658042 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:58.658013    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 11 00:05:58.870096 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:58.870061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:59.558864 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:05:59.558829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:08.990674 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:08.990629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:06:11.351259 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.351202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:11.381431 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.381400    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"]
Sep 11 00:06:11.381734 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.381709    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs" podUID="7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" containerName="payload-processing" containerID="cri-o://652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6" gracePeriod=30
Sep 11 00:06:11.415386 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.415344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:11.475229 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.475198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:11.556341 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.556198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:11.610196 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.610162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:11.626948 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.626924    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:06:11.702873 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.702840    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-plugins-config-volume\") pod \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") "
Sep 11 00:06:11.703050 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.702946    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbtk2\" (UniqueName: \"kubernetes.io/projected/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-kube-api-access-wbtk2\") pod \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\" (UID: \"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0\") "
Sep 11 00:06:11.703237 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.703212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" (UID: "7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:06:11.705085 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.705061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-kube-api-access-wbtk2" (OuterVolumeSpecName: "kube-api-access-wbtk2") pod "7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" (UID: "7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0"). InnerVolumeSpecName "kube-api-access-wbtk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:06:11.804706 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.804675    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:11.804706 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.804702    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbtk2\" (UniqueName: \"kubernetes.io/projected/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0-kube-api-access-wbtk2\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:11.958606 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:11.958528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:12.086858 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.086821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:12.387488 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.387453    2578 generic.go:356] "Generic (PLEG): container finished" podID="7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" containerID="652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6" exitCode=0
Sep 11 00:06:12.387949 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.387498    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"
Sep 11 00:06:12.387949 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.387544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs" event={"ID":"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0","Type":"ContainerDied","Data":"652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6"}
Sep 11 00:06:12.387949 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.387617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs" event={"ID":"7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0","Type":"ContainerDied","Data":"a86a19cacb93d640403c8e0de2beeaef9231a217b7f23cf96742e46fac4f2e36"}
Sep 11 00:06:12.387949 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.387639    2578 scope.go:117] "RemoveContainer" containerID="652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6"
Sep 11 00:06:12.396686 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.396665    2578 scope.go:117] "RemoveContainer" containerID="652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6"
Sep 11 00:06:12.396954 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:06:12.396933    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6\": container with ID starting with 652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6 not found: ID does not exist" containerID="652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6"
Sep 11 00:06:12.397002 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.396963    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6"} err="failed to get container status \"652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6\": rpc error: code = NotFound desc = could not find container \"652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6\": container with ID starting with 652a9d21e48084eaadc258656c58333e9b70df3ca3c16acaf9497a297dc74bf6 not found: ID does not exist"
Sep 11 00:06:12.409525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.409503    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"]
Sep 11 00:06:12.415012 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.414992    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c3755-58cb685c45-lrjgs"]
Sep 11 00:06:12.534359 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.534330    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" path="/var/lib/kubelet/pods/7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0/volumes"
Sep 11 00:06:12.926856 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:12.926807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:13.710492 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:13.710461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:14.815259 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:14.815228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:16.938618 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:16.938573    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"]
Sep 11 00:06:16.985008 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:16.984972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:21.295706 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:21.295672    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:06:31.958338 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:31.958251    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz" podUID="3c266ab1-f682-4eeb-8c9a-9b059297f1ae" containerName="main" containerID="cri-o://cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6" gracePeriod=45
Sep 11 00:06:32.202018 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.201987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:06:32.289511 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tls-certs\") pod \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") "
Sep 11 00:06:32.289708 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-model-cache\") pod \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") "
Sep 11 00:06:32.289708 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64f48\" (UniqueName: \"kubernetes.io/projected/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-kube-api-access-64f48\") pod \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") "
Sep 11 00:06:32.289708 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-home\") pod \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") "
Sep 11 00:06:32.289708 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-dshm\") pod \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") "
Sep 11 00:06:32.289708 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tmp-dir\") pod \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\" (UID: \"3c266ab1-f682-4eeb-8c9a-9b059297f1ae\") "
Sep 11 00:06:32.289951 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-home" (OuterVolumeSpecName: "home") pod "3c266ab1-f682-4eeb-8c9a-9b059297f1ae" (UID: "3c266ab1-f682-4eeb-8c9a-9b059297f1ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:06:32.289951 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.289892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c266ab1-f682-4eeb-8c9a-9b059297f1ae" (UID: "3c266ab1-f682-4eeb-8c9a-9b059297f1ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:06:32.290055 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.290005    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c266ab1-f682-4eeb-8c9a-9b059297f1ae" (UID: "3c266ab1-f682-4eeb-8c9a-9b059297f1ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:06:32.290112 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.290085    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:32.290112 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.290100    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:32.291816 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.291776    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-dshm" (OuterVolumeSpecName: "dshm") pod "3c266ab1-f682-4eeb-8c9a-9b059297f1ae" (UID: "3c266ab1-f682-4eeb-8c9a-9b059297f1ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:06:32.291816 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.291791    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-kube-api-access-64f48" (OuterVolumeSpecName: "kube-api-access-64f48") pod "3c266ab1-f682-4eeb-8c9a-9b059297f1ae" (UID: "3c266ab1-f682-4eeb-8c9a-9b059297f1ae"). InnerVolumeSpecName "kube-api-access-64f48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:06:32.291816 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.291791    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c266ab1-f682-4eeb-8c9a-9b059297f1ae" (UID: "3c266ab1-f682-4eeb-8c9a-9b059297f1ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:06:32.391269 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.391229    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:32.391269 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.391262    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:32.391502 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.391302    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:32.391502 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.391318    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64f48\" (UniqueName: \"kubernetes.io/projected/3c266ab1-f682-4eeb-8c9a-9b059297f1ae-kube-api-access-64f48\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:06:32.499875 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.499838    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c266ab1-f682-4eeb-8c9a-9b059297f1ae" containerID="cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6" exitCode=0
Sep 11 00:06:32.500075 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.499892    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"
Sep 11 00:06:32.500075 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.499921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz" event={"ID":"3c266ab1-f682-4eeb-8c9a-9b059297f1ae","Type":"ContainerDied","Data":"cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6"}
Sep 11 00:06:32.500075 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.499962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz" event={"ID":"3c266ab1-f682-4eeb-8c9a-9b059297f1ae","Type":"ContainerDied","Data":"55a7a6dc785f2caae60e122665458cb8f86622dc45f799ae84f5f696bb8d3ca0"}
Sep 11 00:06:32.500075 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.499980    2578 scope.go:117] "RemoveContainer" containerID="cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6"
Sep 11 00:06:32.509462 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.509441    2578 scope.go:117] "RemoveContainer" containerID="cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6"
Sep 11 00:06:32.509737 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:06:32.509713    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6\": container with ID starting with cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6 not found: ID does not exist" containerID="cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6"
Sep 11 00:06:32.509807 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.509749    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6"} err="failed to get container status \"cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6\": rpc error: code = NotFound desc = could not find container \"cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6\": container with ID starting with cc3ed1be1b1bcb71ea1247f12ea7e275682ad74066f6dd913eb87ec784e526c6 not found: ID does not exist"
Sep 11 00:06:32.521926 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.521901    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"]
Sep 11 00:06:32.525256 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.525233    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7c3755/ipp-route-7c3755-kserve-64dbfc4995-cbrnz"]
Sep 11 00:06:32.534544 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:32.534521    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c266ab1-f682-4eeb-8c9a-9b059297f1ae" path="/var/lib/kubelet/pods/3c266ab1-f682-4eeb-8c9a-9b059297f1ae/volumes"
Sep 11 00:06:44.776360 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:44.776328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:53.314339 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:53.314305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:54.666669 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:54.666641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:58.828883 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:58.828855    2578 scope.go:117] "RemoveContainer" containerID="c16a516bfc7936d93912d4a68ddf8792e06ee376f395dc869abc5ab464b85eba"
Sep 11 00:06:59.689931 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:06:59.689895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:01.570012 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:01.569959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:13.809262 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:13.809230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:13.873289 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:13.873245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:14.865347 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:14.865306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:14.973203 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:14.973169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:15.779444 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:15.779413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:16.567303 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:16.567257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:17.359515 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:17.359477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:18.180052 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:18.180017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:25.034524 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.034438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:25.038689 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.038657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"]
Sep 11 00:07:25.038985 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.038955    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j" podUID="0558a194-b0ed-4998-b880-dc27fe123758" containerName="payload-processing" containerID="cri-o://1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff" gracePeriod=30
Sep 11 00:07:25.132238 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.132204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:25.229986 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.229951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:25.297679 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.297610    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:07:25.298201 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.298181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:25.380871 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.380846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0558a194-b0ed-4998-b880-dc27fe123758-plugins-config-volume\") pod \"0558a194-b0ed-4998-b880-dc27fe123758\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") "
Sep 11 00:07:25.381026 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.380971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgg4l\" (UniqueName: \"kubernetes.io/projected/0558a194-b0ed-4998-b880-dc27fe123758-kube-api-access-sgg4l\") pod \"0558a194-b0ed-4998-b880-dc27fe123758\" (UID: \"0558a194-b0ed-4998-b880-dc27fe123758\") "
Sep 11 00:07:25.381224 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.381201    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0558a194-b0ed-4998-b880-dc27fe123758-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0558a194-b0ed-4998-b880-dc27fe123758" (UID: "0558a194-b0ed-4998-b880-dc27fe123758"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:07:25.383017 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.382993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0558a194-b0ed-4998-b880-dc27fe123758-kube-api-access-sgg4l" (OuterVolumeSpecName: "kube-api-access-sgg4l") pod "0558a194-b0ed-4998-b880-dc27fe123758" (UID: "0558a194-b0ed-4998-b880-dc27fe123758"). InnerVolumeSpecName "kube-api-access-sgg4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:07:25.482049 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.482014    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgg4l\" (UniqueName: \"kubernetes.io/projected/0558a194-b0ed-4998-b880-dc27fe123758-kube-api-access-sgg4l\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:25.482049 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.482043    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0558a194-b0ed-4998-b880-dc27fe123758-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:25.663307 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.663210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:25.774993 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.774958    2578 generic.go:356] "Generic (PLEG): container finished" podID="0558a194-b0ed-4998-b880-dc27fe123758" containerID="1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff" exitCode=0
Sep 11 00:07:25.775179 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.775008    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"
Sep 11 00:07:25.775179 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.775046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j" event={"ID":"0558a194-b0ed-4998-b880-dc27fe123758","Type":"ContainerDied","Data":"1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff"}
Sep 11 00:07:25.775179 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.775096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j" event={"ID":"0558a194-b0ed-4998-b880-dc27fe123758","Type":"ContainerDied","Data":"7886d4eb0c0684756c840135a52c86afde6947ddcad7570bbb19e3a2d1cfa60d"}
Sep 11 00:07:25.775179 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.775118    2578 scope.go:117] "RemoveContainer" containerID="1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff"
Sep 11 00:07:25.786267 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.786243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:25.787222 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.787208    2578 scope.go:117] "RemoveContainer" containerID="1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff"
Sep 11 00:07:25.787490 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:07:25.787470    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff\": container with ID starting with 1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff not found: ID does not exist" containerID="1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff"
Sep 11 00:07:25.787560 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.787500    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff"} err="failed to get container status \"1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff\": rpc error: code = NotFound desc = could not find container \"1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff\": container with ID starting with 1d24431dc20a91b78f9ad72d4ee2ea9081154969643d4cc74c45e8ed04a56bff not found: ID does not exist"
Sep 11 00:07:25.798579 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.798549    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"]
Sep 11 00:07:25.814103 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:25.814075    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-da6814-6699874d75-kxr4j"]
Sep 11 00:07:26.534309 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:26.534256    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0558a194-b0ed-4998-b880-dc27fe123758" path="/var/lib/kubelet/pods/0558a194-b0ed-4998-b880-dc27fe123758/volumes"
Sep 11 00:07:26.574551 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:26.574519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:27.472464 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:27.472435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:28.285087 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:28.285049    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:07:31.341248 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:31.341210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:31.442478 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:31.442438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:31.600324 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:31.600224    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"]
Sep 11 00:07:31.600510 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:31.600487    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy" containerID="cri-o://b3081405ac09921f312f1f2e4353f6526e8a6f3b5da5838651eef6d1cc00267a" gracePeriod=30
Sep 11 00:07:31.620389 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:31.620362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:32.786733 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:32.786700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:33.573846 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:33.573808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:34.251153 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:34.251117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:36.835544 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.835498    2578 generic.go:356] "Generic (PLEG): container finished" podID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerID="b3081405ac09921f312f1f2e4353f6526e8a6f3b5da5838651eef6d1cc00267a" exitCode=0
Sep 11 00:07:36.835901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.835541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" event={"ID":"220c1333-bb6f-47bf-ac86-883cba46d4fc","Type":"ContainerDied","Data":"b3081405ac09921f312f1f2e4353f6526e8a6f3b5da5838651eef6d1cc00267a"}
Sep 11 00:07:36.835901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.835598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7" event={"ID":"220c1333-bb6f-47bf-ac86-883cba46d4fc","Type":"ContainerDied","Data":"2ab0833856117a2de68c8286ce89cca17e01cca0fb26c818e13fd5bf1b31d6b8"}
Sep 11 00:07:36.835901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.835618    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab0833856117a2de68c8286ce89cca17e01cca0fb26c818e13fd5bf1b31d6b8"
Sep 11 00:07:36.835901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.835632    2578 scope.go:117] "RemoveContainer" containerID="4820779e0296a3aeeae592f552550c6d6b7e5ec2ec4b732933ed4071987a7664"
Sep 11 00:07:36.847491 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.847470    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:07:36.990861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.990776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-socket\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.990861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.990813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-credential-socket\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.990861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.990833    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk8hd\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-kube-api-access-kk8hd\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.991126 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.990880    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220c1333-bb6f-47bf-ac86-883cba46d4fc-istiod-ca-cert\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.991126 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.990923    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-data\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.991126 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.990984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-token\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.991126 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-podinfo\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.991126 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:36.991126 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991085    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-certs\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.991483 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:36.991483 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-envoy\") pod \"220c1333-bb6f-47bf-ac86-883cba46d4fc\" (UID: \"220c1333-bb6f-47bf-ac86-883cba46d4fc\") "
Sep 11 00:07:36.991483 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220c1333-bb6f-47bf-ac86-883cba46d4fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:07:36.991483 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:36.991694 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991564    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220c1333-bb6f-47bf-ac86-883cba46d4fc-istiod-ca-cert\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:36.991694 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:36.991694 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991587    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:36.991694 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991601    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-workload-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:36.991694 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.991611    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-credential-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:36.993366 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.993340    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:36.993488 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.993403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 00:07:36.993672 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.993641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:07:36.993672 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:36.993649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-kube-api-access-kk8hd" (OuterVolumeSpecName: "kube-api-access-kk8hd") pod "220c1333-bb6f-47bf-ac86-883cba46d4fc" (UID: "220c1333-bb6f-47bf-ac86-883cba46d4fc"). InnerVolumeSpecName "kube-api-access-kk8hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:07:37.092576 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.092538    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-data\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:37.092576 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.092569    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-token\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:37.092576 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.092580    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-podinfo\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:37.092823 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.092594    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220c1333-bb6f-47bf-ac86-883cba46d4fc-istio-envoy\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:37.092823 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.092605    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8hd\" (UniqueName: \"kubernetes.io/projected/220c1333-bb6f-47bf-ac86-883cba46d4fc-kube-api-access-kk8hd\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:37.842658 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.842626    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"
Sep 11 00:07:37.884041 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.884009    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"]
Sep 11 00:07:37.889228 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:37.889201    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-da6814-openshift-default-796d4bfc4d-54md7"]
Sep 11 00:07:38.533903 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:38.533856    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" path="/var/lib/kubelet/pods/220c1333-bb6f-47bf-ac86-883cba46d4fc/volumes"
Sep 11 00:07:42.508879 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.508846    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"]
Sep 11 00:07:42.514269 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.514238    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6fa41e-29818080-86hpv"]
Sep 11 00:07:42.514470 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.514442    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:42.536092 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.536066    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bca4e72-0757-4243-ad19-9d7ebbfc4564" path="/var/lib/kubelet/pods/6bca4e72-0757-4243-ad19-9d7ebbfc4564/volumes"
Sep 11 00:07:42.536716 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.536690    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"]
Sep 11 00:07:42.536939 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.536913    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b" podUID="3f07e4f5-d797-429c-bac9-91c4b9403aec" containerName="payload-processing" containerID="cri-o://4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6" gracePeriod=30
Sep 11 00:07:42.612318 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.612249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:42.711756 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.711724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:42.786435 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.786384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:42.792589 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.792567    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 11 00:07:42.870294 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.870237    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f07e4f5-d797-429c-bac9-91c4b9403aec" containerID="4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6" exitCode=0
Sep 11 00:07:42.870459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.870296    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"
Sep 11 00:07:42.870459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.870327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b" event={"ID":"3f07e4f5-d797-429c-bac9-91c4b9403aec","Type":"ContainerDied","Data":"4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6"}
Sep 11 00:07:42.870459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.870387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b" event={"ID":"3f07e4f5-d797-429c-bac9-91c4b9403aec","Type":"ContainerDied","Data":"25a184e11a9150933d0f88aaebd6cb2c8e14136f8fef66dd97f91ce20dd1d6fc"}
Sep 11 00:07:42.870459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.870419    2578 scope.go:117] "RemoveContainer" containerID="4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6"
Sep 11 00:07:42.879475 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.879454    2578 scope.go:117] "RemoveContainer" containerID="4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6"
Sep 11 00:07:42.879754 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:07:42.879730    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6\": container with ID starting with 4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6 not found: ID does not exist" containerID="4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6"
Sep 11 00:07:42.879837 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.879771    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6"} err="failed to get container status \"4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6\": rpc error: code = NotFound desc = could not find container \"4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6\": container with ID starting with 4540f1820a6c349c89e9d2a9da29e9751777d552e3ef1d58f596c60e36f694b6 not found: ID does not exist"
Sep 11 00:07:42.948294 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.948242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f07e4f5-d797-429c-bac9-91c4b9403aec-plugins-config-volume\") pod \"3f07e4f5-d797-429c-bac9-91c4b9403aec\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") "
Sep 11 00:07:42.948461 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.948302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r62bn\" (UniqueName: \"kubernetes.io/projected/3f07e4f5-d797-429c-bac9-91c4b9403aec-kube-api-access-r62bn\") pod \"3f07e4f5-d797-429c-bac9-91c4b9403aec\" (UID: \"3f07e4f5-d797-429c-bac9-91c4b9403aec\") "
Sep 11 00:07:42.948622 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.948599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f07e4f5-d797-429c-bac9-91c4b9403aec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f07e4f5-d797-429c-bac9-91c4b9403aec" (UID: "3f07e4f5-d797-429c-bac9-91c4b9403aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:07:42.950563 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:42.950537    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f07e4f5-d797-429c-bac9-91c4b9403aec-kube-api-access-r62bn" (OuterVolumeSpecName: "kube-api-access-r62bn") pod "3f07e4f5-d797-429c-bac9-91c4b9403aec" (UID: "3f07e4f5-d797-429c-bac9-91c4b9403aec"). InnerVolumeSpecName "kube-api-access-r62bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:07:43.049870 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:43.049794    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f07e4f5-d797-429c-bac9-91c4b9403aec-plugins-config-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:43.049870 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:43.049834    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r62bn\" (UniqueName: \"kubernetes.io/projected/3f07e4f5-d797-429c-bac9-91c4b9403aec-kube-api-access-r62bn\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:43.126874 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:43.126831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:43.192675 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:43.192644    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"]
Sep 11 00:07:43.194890 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:43.194840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:43.196918 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:43.196896    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6fa41e-fd44cb69-4ll6b"]
Sep 11 00:07:43.956888 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:43.956856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:44.534868 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:44.534831    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f07e4f5-d797-429c-bac9-91c4b9403aec" path="/var/lib/kubelet/pods/3f07e4f5-d797-429c-bac9-91c4b9403aec/volumes"
Sep 11 00:07:45.397814 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:45.397779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:07:49.099291 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:49.099242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:49.165941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:49.165910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:49.347884 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:49.347853    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"]
Sep 11 00:07:49.348152 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:49.348107    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy" containerID="cri-o://56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569" gracePeriod=30
Sep 11 00:07:49.415217 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:49.415189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:50.177395 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:50.177364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:50.985113 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:50.985076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:51.792537 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:51.792505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:54.598461 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.598437    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 11 00:07:54.653247 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-token\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.653247 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653201    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-data\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.653247 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmkrr\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-kube-api-access-tmkrr\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.653538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-envoy\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.653538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ede721-c87a-419c-89b0-3ef6dbbab219-istiod-ca-cert\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.653538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-certs\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.654095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-credential-socket\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.654095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-podinfo\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.654095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-socket\") pod \"30ede721-c87a-419c-89b0-3ef6dbbab219\" (UID: \"30ede721-c87a-419c-89b0-3ef6dbbab219\") "
Sep 11 00:07:54.654095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653552    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-data" (OuterVolumeSpecName: "istio-data") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:54.654095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:54.654095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653912    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-data\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.654095 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.653961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:54.654487 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.654161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:54.654487 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.654375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ede721-c87a-419c-89b0-3ef6dbbab219-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:07:54.655842 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.655808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-kube-api-access-tmkrr" (OuterVolumeSpecName: "kube-api-access-tmkrr") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "kube-api-access-tmkrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:07:54.655969 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.655920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:07:54.656148 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.656121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-token" (OuterVolumeSpecName: "istio-token") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:07:54.656388 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.656363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30ede721-c87a-419c-89b0-3ef6dbbab219" (UID: "30ede721-c87a-419c-89b0-3ef6dbbab219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 00:07:54.754757 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754725    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-credential-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.754757 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754753    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-podinfo\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.754941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754767    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-socket\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.754941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754778    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-token\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.754941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754789    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkrr\" (UniqueName: \"kubernetes.io/projected/30ede721-c87a-419c-89b0-3ef6dbbab219-kube-api-access-tmkrr\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.754941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754801    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-istio-envoy\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.754941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754812    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ede721-c87a-419c-89b0-3ef6dbbab219-istiod-ca-cert\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.754941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.754824    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ede721-c87a-419c-89b0-3ef6dbbab219-workload-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:07:54.928990 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.928902    2578 generic.go:356] "Generic (PLEG): container finished" podID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerID="56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569" exitCode=0
Sep 11 00:07:54.928990 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.928949    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"
Sep 11 00:07:54.929166 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.928990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" event={"ID":"30ede721-c87a-419c-89b0-3ef6dbbab219","Type":"ContainerDied","Data":"56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569"}
Sep 11 00:07:54.929166 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.929042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs" event={"ID":"30ede721-c87a-419c-89b0-3ef6dbbab219","Type":"ContainerDied","Data":"a74f655b6745d98fe1e75806365895ecac21efe6b95f7e65626de18e0bad5d40"}
Sep 11 00:07:54.929166 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.929064    2578 scope.go:117] "RemoveContainer" containerID="56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569"
Sep 11 00:07:54.938861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.938843    2578 scope.go:117] "RemoveContainer" containerID="4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80"
Sep 11 00:07:54.946475 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.946459    2578 scope.go:117] "RemoveContainer" containerID="56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569"
Sep 11 00:07:54.946730 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:07:54.946709    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569\": container with ID starting with 56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569 not found: ID does not exist" containerID="56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569"
Sep 11 00:07:54.946795 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.946738    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569"} err="failed to get container status \"56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569\": rpc error: code = NotFound desc = could not find container \"56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569\": container with ID starting with 56c0a0c3b8358652ec4ba188c1b83cfbdb7e74a9742e90309ccb3ea546faf569 not found: ID does not exist"
Sep 11 00:07:54.946795 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.946754    2578 scope.go:117] "RemoveContainer" containerID="4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80"
Sep 11 00:07:54.946999 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:07:54.946982    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80\": container with ID starting with 4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80 not found: ID does not exist" containerID="4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80"
Sep 11 00:07:54.947036 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.947006    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80"} err="failed to get container status \"4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80\": rpc error: code = NotFound desc = could not find container \"4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80\": container with ID starting with 4669e9cf37c8364414aec6fec0e028551279b287ff426b100eb74ac267c1cc80 not found: ID does not exist"
Sep 11 00:07:54.953632 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.953610    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"]
Sep 11 00:07:54.960020 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:54.959995    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6fa41e-openshift-default-56dd9588bb-5wnhs"]
Sep 11 00:07:56.534362 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:56.534332    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" path="/var/lib/kubelet/pods/30ede721-c87a-419c-89b0-3ef6dbbab219/volumes"
Sep 11 00:07:58.637079 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:58.637050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:58.859345 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:07:58.859312    2578 scope.go:117] "RemoveContainer" containerID="5fcc10537924dc12c75fa51b6f49c8d892c1642c7bd7094534685e5f7ba8aa43"
Sep 11 00:08:00.181760 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.181724    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"]
Sep 11 00:08:00.182197 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.182039    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l" podUID="577e0dfc-7d72-47e5-94f9-c8db6c66ef39" containerName="maas-api" containerID="cri-o://936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6" gracePeriod=30
Sep 11 00:08:00.209776 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.209743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:00.214041 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.214005    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"]
Sep 11 00:08:00.215613 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.215589    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-37c598-29818080-mdhvs"]
Sep 11 00:08:00.309531 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.309494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:00.407566 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.407536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:00.424090 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.424059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 11 00:08:00.489594 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.489523    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:00.504506 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.504480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-metrics-certs\") pod \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") "
Sep 11 00:08:00.504627 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.504535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmkvl\" (UniqueName: \"kubernetes.io/projected/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-kube-api-access-cmkvl\") pod \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") "
Sep 11 00:08:00.504627 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.504582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-openshift-service-ca\") pod \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") "
Sep 11 00:08:00.504627 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.504608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-maas-api-tls\") pod \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\" (UID: \"577e0dfc-7d72-47e5-94f9-c8db6c66ef39\") "
Sep 11 00:08:00.504929 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.504897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "577e0dfc-7d72-47e5-94f9-c8db6c66ef39" (UID: "577e0dfc-7d72-47e5-94f9-c8db6c66ef39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:08:00.507112 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.507081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "577e0dfc-7d72-47e5-94f9-c8db6c66ef39" (UID: "577e0dfc-7d72-47e5-94f9-c8db6c66ef39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:08:00.507230 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.507120    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-kube-api-access-cmkvl" (OuterVolumeSpecName: "kube-api-access-cmkvl") pod "577e0dfc-7d72-47e5-94f9-c8db6c66ef39" (UID: "577e0dfc-7d72-47e5-94f9-c8db6c66ef39"). InnerVolumeSpecName "kube-api-access-cmkvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:08:00.507230 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.507180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "577e0dfc-7d72-47e5-94f9-c8db6c66ef39" (UID: "577e0dfc-7d72-47e5-94f9-c8db6c66ef39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:08:00.534329 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.534305    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5baf9350-0ee2-4a52-93eb-c822f85870e8" path="/var/lib/kubelet/pods/5baf9350-0ee2-4a52-93eb-c822f85870e8/volumes"
Sep 11 00:08:00.606174 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.606033    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-metrics-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:08:00.606174 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.606067    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkvl\" (UniqueName: \"kubernetes.io/projected/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-kube-api-access-cmkvl\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:08:00.606174 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.606103    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-openshift-service-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:08:00.606174 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.606131    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/577e0dfc-7d72-47e5-94f9-c8db6c66ef39-maas-api-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:08:00.766589 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.766549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:00.875537 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.875501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:00.960329 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.960295    2578 generic.go:356] "Generic (PLEG): container finished" podID="577e0dfc-7d72-47e5-94f9-c8db6c66ef39" containerID="936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6" exitCode=0
Sep 11 00:08:00.960507 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.960333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l" event={"ID":"577e0dfc-7d72-47e5-94f9-c8db6c66ef39","Type":"ContainerDied","Data":"936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6"}
Sep 11 00:08:00.960507 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.960387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l" event={"ID":"577e0dfc-7d72-47e5-94f9-c8db6c66ef39","Type":"ContainerDied","Data":"0ae6a78a3756ed2e08f73bae7a205df0477ad75d0a251dff464f5dc588c5a9ad"}
Sep 11 00:08:00.960507 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.960409    2578 scope.go:117] "RemoveContainer" containerID="936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6"
Sep 11 00:08:00.960507 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.960349    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"
Sep 11 00:08:00.969050 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.969029    2578 scope.go:117] "RemoveContainer" containerID="936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6"
Sep 11 00:08:00.969345 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:08:00.969322    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6\": container with ID starting with 936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6 not found: ID does not exist" containerID="936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6"
Sep 11 00:08:00.969412 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.969358    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6"} err="failed to get container status \"936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6\": rpc error: code = NotFound desc = could not find container \"936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6\": container with ID starting with 936e553e1dc70cf6190089a5b8261137108077134fc4272d5a26fc7907878de6 not found: ID does not exist"
Sep 11 00:08:00.977029 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.977007    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"]
Sep 11 00:08:00.981758 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:00.981737    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-37c598-cb97bd7cb-n8x9l"]
Sep 11 00:08:02.533653 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:02.533621    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="577e0dfc-7d72-47e5-94f9-c8db6c66ef39" path="/var/lib/kubelet/pods/577e0dfc-7d72-47e5-94f9-c8db6c66ef39/volumes"
Sep 11 00:08:02.789459 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:02.789371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:08:06.810406 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:06.810370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:06.908600 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:06.908564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:07.644497 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:07.644459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:11.982304 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:11.982258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:08:28.101229 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:28.101191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:08:58.888749 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:08:58.888660    2578 scope.go:117] "RemoveContainer" containerID="77fbe90e47e6726b4fd89204545b4f8871370b229aab1437fb356bba691e1da0"
Sep 11 00:09:33.186986 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:09:33.186943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:09:33.264826 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:09:33.264784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:09:34.883922 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:09:34.883886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:09:54.133505 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:09:54.133468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:09:54.234121 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:09:54.234083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:09:58.989944 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:09:58.989907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:05.052185 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:05.052145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:10:05.115641 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:05.115602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:10:06.493936 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:06.493897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:13.688921 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:13.688884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:19.783601 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:19.783560    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:28.510124 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:28.510045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:32.384492 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:32.384456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:39.993468 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:39.993436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:48.482195 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:48.482152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:10:57.349456 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.349416    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"]
Sep 11 00:10:57.349951 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.349694    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd" podUID="8600967b-cbb0-4d07-87e0-fa266b3c4f11" containerName="manager" containerID="cri-o://58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd" gracePeriod=10
Sep 11 00:10:57.598193 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.598162    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 11 00:10:57.709640 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.709555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcj2k\" (UniqueName: \"kubernetes.io/projected/8600967b-cbb0-4d07-87e0-fa266b3c4f11-kube-api-access-fcj2k\") pod \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") "
Sep 11 00:10:57.709794 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.709660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8600967b-cbb0-4d07-87e0-fa266b3c4f11-extensions-socket-volume\") pod \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\" (UID: \"8600967b-cbb0-4d07-87e0-fa266b3c4f11\") "
Sep 11 00:10:57.710072 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.710049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8600967b-cbb0-4d07-87e0-fa266b3c4f11-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8600967b-cbb0-4d07-87e0-fa266b3c4f11" (UID: "8600967b-cbb0-4d07-87e0-fa266b3c4f11"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:10:57.711774 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.711747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8600967b-cbb0-4d07-87e0-fa266b3c4f11-kube-api-access-fcj2k" (OuterVolumeSpecName: "kube-api-access-fcj2k") pod "8600967b-cbb0-4d07-87e0-fa266b3c4f11" (UID: "8600967b-cbb0-4d07-87e0-fa266b3c4f11"). InnerVolumeSpecName "kube-api-access-fcj2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:10:57.770302 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.770251    2578 generic.go:356] "Generic (PLEG): container finished" podID="8600967b-cbb0-4d07-87e0-fa266b3c4f11" containerID="58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd" exitCode=0
Sep 11 00:10:57.770499 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.770321    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"
Sep 11 00:10:57.770499 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.770343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd" event={"ID":"8600967b-cbb0-4d07-87e0-fa266b3c4f11","Type":"ContainerDied","Data":"58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd"}
Sep 11 00:10:57.770499 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.770401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd" event={"ID":"8600967b-cbb0-4d07-87e0-fa266b3c4f11","Type":"ContainerDied","Data":"24a80d558165ef815f02fee626c1273ef6fa9ec3ea8f573b13349d075a4d9381"}
Sep 11 00:10:57.770499 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.770423    2578 scope.go:117] "RemoveContainer" containerID="58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd"
Sep 11 00:10:57.784602 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.784582    2578 scope.go:117] "RemoveContainer" containerID="58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd"
Sep 11 00:10:57.784899 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:10:57.784882    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd\": container with ID starting with 58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd not found: ID does not exist" containerID="58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd"
Sep 11 00:10:57.784941 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.784906    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd"} err="failed to get container status \"58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd\": rpc error: code = NotFound desc = could not find container \"58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd\": container with ID starting with 58c2fe8d8e69cdb066aed5f72ffd38a3e881a52e8bf68176f446235315832acd not found: ID does not exist"
Sep 11 00:10:57.797306 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.797259    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"]
Sep 11 00:10:57.800960 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.800938    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l58rd"]
Sep 11 00:10:57.810660 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.810640    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8600967b-cbb0-4d07-87e0-fa266b3c4f11-extensions-socket-volume\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:10:57.810746 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:57.810662    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcj2k\" (UniqueName: \"kubernetes.io/projected/8600967b-cbb0-4d07-87e0-fa266b3c4f11-kube-api-access-fcj2k\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:10:58.534132 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:58.534094    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8600967b-cbb0-4d07-87e0-fa266b3c4f11" path="/var/lib/kubelet/pods/8600967b-cbb0-4d07-87e0-fa266b3c4f11/volumes"
Sep 11 00:10:58.683083 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:58.683058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 11 00:10:58.686112 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:58.686090    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 11 00:10:58.916157 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:10:58.916070    2578 scope.go:117] "RemoveContainer" containerID="b3081405ac09921f312f1f2e4353f6526e8a6f3b5da5838651eef6d1cc00267a"
Sep 11 00:11:07.990361 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:07.990264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:11:58.046429 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046344    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"]
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046699    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046712    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046720    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0558a194-b0ed-4998-b880-dc27fe123758" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046726    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0558a194-b0ed-4998-b880-dc27fe123758" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046734    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c266ab1-f682-4eeb-8c9a-9b059297f1ae" containerName="main"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046742    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c266ab1-f682-4eeb-8c9a-9b059297f1ae" containerName="main"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046750    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577e0dfc-7d72-47e5-94f9-c8db6c66ef39" containerName="maas-api"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046756    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="577e0dfc-7d72-47e5-94f9-c8db6c66ef39" containerName="maas-api"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046763    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046768    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046775    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f07e4f5-d797-429c-bac9-91c4b9403aec" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f07e4f5-d797-429c-bac9-91c4b9403aec" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046785    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046790    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046798    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046803    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046809    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f1ecf8-943a-4833-af82-4fada16c7af5" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046814    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f1ecf8-943a-4833-af82-4fada16c7af5" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046822    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046827    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046844    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8600967b-cbb0-4d07-87e0-fa266b3c4f11" containerName="manager"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046850    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8600967b-cbb0-4d07-87e0-fa266b3c4f11" containerName="manager"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046898    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f07e4f5-d797-429c-bac9-91c4b9403aec" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046905    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046911    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c266ab1-f682-4eeb-8c9a-9b059297f1ae" containerName="main"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046917    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="577e0dfc-7d72-47e5-94f9-c8db6c66ef39" containerName="maas-api"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046923    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0558a194-b0ed-4998-b880-dc27fe123758" containerName="payload-processing"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046928    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy"
Sep 11 00:11:58.046919 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046933    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd7a5eb-84df-4f3a-9efd-f6f8e5caa5c0" containerName="payload-processing"
Sep 11 00:11:58.047850 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046939    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f1ecf8-943a-4833-af82-4fada16c7af5" containerName="payload-processing"
Sep 11 00:11:58.047850 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.046946    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8600967b-cbb0-4d07-87e0-fa266b3c4f11" containerName="manager"
Sep 11 00:11:58.050082 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.050064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.053524 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.053497    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fwgcp\""
Sep 11 00:11:58.062652 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.062625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"]
Sep 11 00:11:58.129425 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.129381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzdf\" (UniqueName: \"kubernetes.io/projected/98e883ef-8d64-4061-830a-f068947f788f-kube-api-access-nfzdf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s69jt\" (UID: \"98e883ef-8d64-4061-830a-f068947f788f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.129614 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.129515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e883ef-8d64-4061-830a-f068947f788f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s69jt\" (UID: \"98e883ef-8d64-4061-830a-f068947f788f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.230799 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.230755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzdf\" (UniqueName: \"kubernetes.io/projected/98e883ef-8d64-4061-830a-f068947f788f-kube-api-access-nfzdf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s69jt\" (UID: \"98e883ef-8d64-4061-830a-f068947f788f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.230993 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.230863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e883ef-8d64-4061-830a-f068947f788f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s69jt\" (UID: \"98e883ef-8d64-4061-830a-f068947f788f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.231319 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.231262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e883ef-8d64-4061-830a-f068947f788f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s69jt\" (UID: \"98e883ef-8d64-4061-830a-f068947f788f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.241365 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.241339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzdf\" (UniqueName: \"kubernetes.io/projected/98e883ef-8d64-4061-830a-f068947f788f-kube-api-access-nfzdf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s69jt\" (UID: \"98e883ef-8d64-4061-830a-f068947f788f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.359674 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.359579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:58.635846 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:58.635819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"]
Sep 11 00:11:58.638562 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:11:58.638534    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e883ef_8d64_4061_830a_f068947f788f.slice/crio-5ecc0cef96b942e560121911d1d1b7e6bea6c6e506ea08d1bb36d9024f9c20be WatchSource:0}: Error finding container 5ecc0cef96b942e560121911d1d1b7e6bea6c6e506ea08d1bb36d9024f9c20be: Status 404 returned error can't find the container with id 5ecc0cef96b942e560121911d1d1b7e6bea6c6e506ea08d1bb36d9024f9c20be
Sep 11 00:11:59.040910 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:59.040873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt" event={"ID":"98e883ef-8d64-4061-830a-f068947f788f","Type":"ContainerStarted","Data":"851a524d7a5b5ca0c5ad92a963466547255d312ef14fd15285a0cf016094b7cc"}
Sep 11 00:11:59.041096 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:59.040931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt" event={"ID":"98e883ef-8d64-4061-830a-f068947f788f","Type":"ContainerStarted","Data":"5ecc0cef96b942e560121911d1d1b7e6bea6c6e506ea08d1bb36d9024f9c20be"}
Sep 11 00:11:59.041096 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:59.040954    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:11:59.062946 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:11:59.062899    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt" podStartSLOduration=1.062883779 podStartE2EDuration="1.062883779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:11:59.061745029 +0000 UTC m=+2161.017976832" watchObservedRunningTime="2026-09-11 00:11:59.062883779 +0000 UTC m=+2161.019115582"
Sep 11 00:12:03.633435 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:03.633396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:12:04.898269 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:04.898233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:12:10.048321 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:10.048268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s69jt"
Sep 11 00:12:19.895589 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:19.895545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:12:27.880701 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:27.880663    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:12:36.290965 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:36.290928    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:12:44.131407 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.131373    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"]
Sep 11 00:12:44.131817 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.131802    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ede721-c87a-419c-89b0-3ef6dbbab219" containerName="istio-proxy"
Sep 11 00:12:44.131868 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.131820    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="220c1333-bb6f-47bf-ac86-883cba46d4fc" containerName="istio-proxy"
Sep 11 00:12:44.134940 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.134918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.138481 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.138458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ff855ca8-kserve-self-signed-certs\""
Sep 11 00:12:44.145700 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.145675    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"]
Sep 11 00:12:44.168584 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.168550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:12:44.225668 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.225632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-tmp-dir\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.225831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.225671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-dshm\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.225831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.225695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b151d92-bde0-472b-a4b8-3a889a9f5124-tls-certs\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.225831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.225758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmf24\" (UniqueName: \"kubernetes.io/projected/6b151d92-bde0-472b-a4b8-3a889a9f5124-kube-api-access-xmf24\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.225831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.225818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-model-cache\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.226013 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.225844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-home\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326336 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-home\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326539 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-tmp-dir\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326539 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-dshm\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326539 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b151d92-bde0-472b-a4b8-3a889a9f5124-tls-certs\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326539 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmf24\" (UniqueName: \"kubernetes.io/projected/6b151d92-bde0-472b-a4b8-3a889a9f5124-kube-api-access-xmf24\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326539 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-model-cache\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326816 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-home\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326875 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-tmp-dir\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.326918 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.326883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-model-cache\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.328802 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.328778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-dshm\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.328907 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.328888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b151d92-bde0-472b-a4b8-3a889a9f5124-tls-certs\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.334872 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.334849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmf24\" (UniqueName: \"kubernetes.io/projected/6b151d92-bde0-472b-a4b8-3a889a9f5124-kube-api-access-xmf24\") pod \"e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") " pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.445504 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.445414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:44.716844 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:44.716817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"]
Sep 11 00:12:44.719387 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:12:44.719351    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b151d92_bde0_472b_a4b8_3a889a9f5124.slice/crio-4055c33f4d60d144639ce6d80a5f4409b368f7c8e2c5956e2f3bc3aeef3e3ccd WatchSource:0}: Error finding container 4055c33f4d60d144639ce6d80a5f4409b368f7c8e2c5956e2f3bc3aeef3e3ccd: Status 404 returned error can't find the container with id 4055c33f4d60d144639ce6d80a5f4409b368f7c8e2c5956e2f3bc3aeef3e3ccd
Sep 11 00:12:45.253667 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:45.253627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2" event={"ID":"6b151d92-bde0-472b-a4b8-3a889a9f5124","Type":"ContainerStarted","Data":"382124f8a5a5dc8376b13c2838e0528ed7a636c290bc4b1482c23993317d2ccf"}
Sep 11 00:12:45.253667 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:45.253670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2" event={"ID":"6b151d92-bde0-472b-a4b8-3a889a9f5124","Type":"ContainerStarted","Data":"4055c33f4d60d144639ce6d80a5f4409b368f7c8e2c5956e2f3bc3aeef3e3ccd"}
Sep 11 00:12:45.271839 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:45.271775    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2" podStartSLOduration=1.271757024 podStartE2EDuration="1.271757024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:12:45.270257034 +0000 UTC m=+2207.226488849" watchObservedRunningTime="2026-09-11 00:12:45.271757024 +0000 UTC m=+2207.227989036"
Sep 11 00:12:46.488061 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:46.488028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:12:50.737050 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.737017    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"]
Sep 11 00:12:50.740391 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.740372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.743122 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.743096    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ff855ca8-kserve-self-signed-certs\""
Sep 11 00:12:50.749737 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.749715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"]
Sep 11 00:12:50.774372 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.774348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:12:50.885551 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.885520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-dshm\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.885721 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.885556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tls-certs\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.885721 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.885642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-home\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.885721 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.885695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tmp-dir\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.885892 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.885772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m58c\" (UniqueName: \"kubernetes.io/projected/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-kube-api-access-6m58c\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.885892 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.885801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-model-cache\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.986610 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.986583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-home\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.986720 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.986616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tmp-dir\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.986720 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.986654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m58c\" (UniqueName: \"kubernetes.io/projected/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-kube-api-access-6m58c\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.986720 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.986671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-model-cache\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.986720 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.986697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-dshm\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.986720 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.986715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tls-certs\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.987043 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.987022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-home\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.987111 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.987039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-model-cache\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.987111 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.987088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tmp-dir\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.989025 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.989007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-dshm\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.989194 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.989179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tls-certs\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:50.995554 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:50.995530    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m58c\" (UniqueName: \"kubernetes.io/projected/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-kube-api-access-6m58c\") pod \"e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") " pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:51.052503 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:51.052480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:12:51.324935 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:51.324906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"]
Sep 11 00:12:51.327231 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:12:51.327207    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a6ca49f_e16d_48f8_9ca1_f42f464e23bf.slice/crio-cb387444fd3ddffd70ead062d0e66b4fae410dade96b0b9e76c31e123427ce43 WatchSource:0}: Error finding container cb387444fd3ddffd70ead062d0e66b4fae410dade96b0b9e76c31e123427ce43: Status 404 returned error can't find the container with id cb387444fd3ddffd70ead062d0e66b4fae410dade96b0b9e76c31e123427ce43
Sep 11 00:12:52.285956 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:52.285925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8" event={"ID":"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf","Type":"ContainerStarted","Data":"9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3"}
Sep 11 00:12:52.286341 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:52.285966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8" event={"ID":"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf","Type":"ContainerStarted","Data":"cb387444fd3ddffd70ead062d0e66b4fae410dade96b0b9e76c31e123427ce43"}
Sep 11 00:12:52.302264 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:52.302224    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8" podStartSLOduration=2.302210357 podStartE2EDuration="2.302210357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:12:52.301124805 +0000 UTC m=+2214.257356619" watchObservedRunningTime="2026-09-11 00:12:52.302210357 +0000 UTC m=+2214.258442160"
Sep 11 00:12:53.196647 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:53.196609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:12:54.445555 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:54.445518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:54.445996 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:54.445571    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:54.458859 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:54.458832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:12:55.317152 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:12:55.317120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:13:01.053061 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:01.053017    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:13:01.053526 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:01.053108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:13:01.065373 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:01.065349    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:13:01.344191 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:01.344099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:13:28.336228 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:28.336142    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"]
Sep 11 00:13:28.352417 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:28.352386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:13:30.046499 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:30.046461    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"]
Sep 11 00:13:30.065222 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:30.065194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:13:30.597166 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:30.597128    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:13:37.791183 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:37.791148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:13:43.356316 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.356230    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2" podUID="6b151d92-bde0-472b-a4b8-3a889a9f5124" containerName="main" containerID="cri-o://382124f8a5a5dc8376b13c2838e0528ed7a636c290bc4b1482c23993317d2ccf" gracePeriod=45
Sep 11 00:13:43.531973 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.531935    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b151d92-bde0-472b-a4b8-3a889a9f5124" containerID="382124f8a5a5dc8376b13c2838e0528ed7a636c290bc4b1482c23993317d2ccf" exitCode=0
Sep 11 00:13:43.532143 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.532010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2" event={"ID":"6b151d92-bde0-472b-a4b8-3a889a9f5124","Type":"ContainerDied","Data":"382124f8a5a5dc8376b13c2838e0528ed7a636c290bc4b1482c23993317d2ccf"}
Sep 11 00:13:43.601465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.601435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:13:43.750048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.749953    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-dshm\") pod \"6b151d92-bde0-472b-a4b8-3a889a9f5124\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") "
Sep 11 00:13:43.750048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b151d92-bde0-472b-a4b8-3a889a9f5124-tls-certs\") pod \"6b151d92-bde0-472b-a4b8-3a889a9f5124\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") "
Sep 11 00:13:43.750048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-home\") pod \"6b151d92-bde0-472b-a4b8-3a889a9f5124\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") "
Sep 11 00:13:43.750377 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmf24\" (UniqueName: \"kubernetes.io/projected/6b151d92-bde0-472b-a4b8-3a889a9f5124-kube-api-access-xmf24\") pod \"6b151d92-bde0-472b-a4b8-3a889a9f5124\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") "
Sep 11 00:13:43.750377 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-model-cache\") pod \"6b151d92-bde0-472b-a4b8-3a889a9f5124\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") "
Sep 11 00:13:43.750377 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-tmp-dir\") pod \"6b151d92-bde0-472b-a4b8-3a889a9f5124\" (UID: \"6b151d92-bde0-472b-a4b8-3a889a9f5124\") "
Sep 11 00:13:43.750377 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-home" (OuterVolumeSpecName: "home") pod "6b151d92-bde0-472b-a4b8-3a889a9f5124" (UID: "6b151d92-bde0-472b-a4b8-3a889a9f5124"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:43.750595 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750494    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:43.750595 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b151d92-bde0-472b-a4b8-3a889a9f5124" (UID: "6b151d92-bde0-472b-a4b8-3a889a9f5124"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:43.750595 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.750510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b151d92-bde0-472b-a4b8-3a889a9f5124" (UID: "6b151d92-bde0-472b-a4b8-3a889a9f5124"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:43.752199 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.752172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-dshm" (OuterVolumeSpecName: "dshm") pod "6b151d92-bde0-472b-a4b8-3a889a9f5124" (UID: "6b151d92-bde0-472b-a4b8-3a889a9f5124"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:43.752340 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.752195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b151d92-bde0-472b-a4b8-3a889a9f5124-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b151d92-bde0-472b-a4b8-3a889a9f5124" (UID: "6b151d92-bde0-472b-a4b8-3a889a9f5124"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:13:43.752340 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.752250    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b151d92-bde0-472b-a4b8-3a889a9f5124-kube-api-access-xmf24" (OuterVolumeSpecName: "kube-api-access-xmf24") pod "6b151d92-bde0-472b-a4b8-3a889a9f5124" (UID: "6b151d92-bde0-472b-a4b8-3a889a9f5124"). InnerVolumeSpecName "kube-api-access-xmf24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:13:43.851164 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.851130    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmf24\" (UniqueName: \"kubernetes.io/projected/6b151d92-bde0-472b-a4b8-3a889a9f5124-kube-api-access-xmf24\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:43.851164 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.851159    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:43.851434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.851188    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:43.851434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.851196    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b151d92-bde0-472b-a4b8-3a889a9f5124-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:43.851434 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:43.851203    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b151d92-bde0-472b-a4b8-3a889a9f5124-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:44.538016 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:44.537986    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"
Sep 11 00:13:44.538016 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:44.538010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2" event={"ID":"6b151d92-bde0-472b-a4b8-3a889a9f5124","Type":"ContainerDied","Data":"4055c33f4d60d144639ce6d80a5f4409b368f7c8e2c5956e2f3bc3aeef3e3ccd"}
Sep 11 00:13:44.538538 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:44.538052    2578 scope.go:117] "RemoveContainer" containerID="382124f8a5a5dc8376b13c2838e0528ed7a636c290bc4b1482c23993317d2ccf"
Sep 11 00:13:44.557639 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:44.557614    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"]
Sep 11 00:13:44.562580 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:44.562555    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ff855ca8-kserve-79c54cb74c-fsjz2"]
Sep 11 00:13:45.066179 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.066133    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8" podUID="7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" containerName="main" containerID="cri-o://9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3" gracePeriod=45
Sep 11 00:13:45.314987 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.314961    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:13:45.464638 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.464548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-dshm\") pod \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") "
Sep 11 00:13:45.464638 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.464611    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tmp-dir\") pod \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") "
Sep 11 00:13:45.464861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.464645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m58c\" (UniqueName: \"kubernetes.io/projected/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-kube-api-access-6m58c\") pod \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") "
Sep 11 00:13:45.464861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.464710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tls-certs\") pod \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") "
Sep 11 00:13:45.464861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.464733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-home\") pod \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") "
Sep 11 00:13:45.464861 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.464803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-model-cache\") pod \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\" (UID: \"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf\") "
Sep 11 00:13:45.465048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.464919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" (UID: "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:45.465048 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.465000    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-home" (OuterVolumeSpecName: "home") pod "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" (UID: "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:45.465156 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.465044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" (UID: "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:45.465156 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.465126    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tmp-dir\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:45.465156 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.465140    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-home\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:45.465156 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.465149    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-model-cache\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:45.466636 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.466612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-dshm" (OuterVolumeSpecName: "dshm") pod "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" (UID: "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:13:45.466751 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.466704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" (UID: "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:13:45.466855 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.466838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-kube-api-access-6m58c" (OuterVolumeSpecName: "kube-api-access-6m58c") pod "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" (UID: "7a6ca49f-e16d-48f8-9ca1-f42f464e23bf"). InnerVolumeSpecName "kube-api-access-6m58c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:13:45.544783 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.544752    2578 generic.go:356] "Generic (PLEG): container finished" podID="7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" containerID="9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3" exitCode=0
Sep 11 00:13:45.545253 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.544804    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"
Sep 11 00:13:45.545253 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.544816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8" event={"ID":"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf","Type":"ContainerDied","Data":"9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3"}
Sep 11 00:13:45.545253 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.544847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8" event={"ID":"7a6ca49f-e16d-48f8-9ca1-f42f464e23bf","Type":"ContainerDied","Data":"cb387444fd3ddffd70ead062d0e66b4fae410dade96b0b9e76c31e123427ce43"}
Sep 11 00:13:45.545253 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.544864    2578 scope.go:117] "RemoveContainer" containerID="9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3"
Sep 11 00:13:45.554345 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.554321    2578 scope.go:117] "RemoveContainer" containerID="9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3"
Sep 11 00:13:45.554655 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:13:45.554636    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3\": container with ID starting with 9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3 not found: ID does not exist" containerID="9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3"
Sep 11 00:13:45.554705 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.554664    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3"} err="failed to get container status \"9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3\": rpc error: code = NotFound desc = could not find container \"9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3\": container with ID starting with 9121b7e4e524c2c094b1338ff3562f57e3019710e09cf4a96c4c2fecfe5791b3 not found: ID does not exist"
Sep 11 00:13:45.565654 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.565631    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-dshm\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:45.565654 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.565654    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m58c\" (UniqueName: \"kubernetes.io/projected/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-kube-api-access-6m58c\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:45.565794 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.565665    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf-tls-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:45.566024 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.566004    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"]
Sep 11 00:13:45.569891 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:45.569862    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ff855ca8-kserve-f46df97dc-2jgw8"]
Sep 11 00:13:46.533462 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:46.533415    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b151d92-bde0-472b-a4b8-3a889a9f5124" path="/var/lib/kubelet/pods/6b151d92-bde0-472b-a4b8-3a889a9f5124/volumes"
Sep 11 00:13:46.533921 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:46.533899    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" path="/var/lib/kubelet/pods/7a6ca49f-e16d-48f8-9ca1-f42f464e23bf/volumes"
Sep 11 00:13:47.488500 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.488455    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:13:47.548619 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.548576    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"]
Sep 11 00:13:47.549125 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.549106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b151d92-bde0-472b-a4b8-3a889a9f5124" containerName="main"
Sep 11 00:13:47.549176 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.549128    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b151d92-bde0-472b-a4b8-3a889a9f5124" containerName="main"
Sep 11 00:13:47.549176 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.549147    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" containerName="main"
Sep 11 00:13:47.549176 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.549156    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" containerName="main"
Sep 11 00:13:47.549302 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.549243    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6ca49f-e16d-48f8-9ca1-f42f464e23bf" containerName="main"
Sep 11 00:13:47.549302 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.549257    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b151d92-bde0-472b-a4b8-3a889a9f5124" containerName="main"
Sep 11 00:13:47.553556 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.553534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.556130 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.556105    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:13:47.557621 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.557595    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 00:13:47.557831 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.557797    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 00:13:47.557944 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.557628    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-78nn9\""
Sep 11 00:13:47.557944 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.557715    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 00:13:47.560983 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.560962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"]
Sep 11 00:13:47.563531 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.563510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-778c967957-c5rrn"]
Sep 11 00:13:47.567179 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.567156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.580945 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.580913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-778c967957-c5rrn"]
Sep 11 00:13:47.686540 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2kw\" (UniqueName: \"kubernetes.io/projected/e652e62b-617d-42e8-9863-659445f40674-kube-api-access-gz2kw\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.686540 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea7640-b694-47b4-8881-28ae3c025d10-openshift-service-ca\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.686833 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e652e62b-617d-42e8-9863-659445f40674-maas-api-tls\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.686833 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-maas-api-tls\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.686833 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxnmr\" (UniqueName: \"kubernetes.io/projected/5bea7640-b694-47b4-8881-28ae3c025d10-kube-api-access-kxnmr\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.686833 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-metrics-certs\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.687029 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e652e62b-617d-42e8-9863-659445f40674-openshift-service-ca\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.687029 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.686902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e652e62b-617d-42e8-9863-659445f40674-metrics-certs\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.788408 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e652e62b-617d-42e8-9863-659445f40674-openshift-service-ca\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.788408 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e652e62b-617d-42e8-9863-659445f40674-metrics-certs\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.788666 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2kw\" (UniqueName: \"kubernetes.io/projected/e652e62b-617d-42e8-9863-659445f40674-kube-api-access-gz2kw\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.788666 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea7640-b694-47b4-8881-28ae3c025d10-openshift-service-ca\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.788666 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e652e62b-617d-42e8-9863-659445f40674-maas-api-tls\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.788826 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-maas-api-tls\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.788826 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxnmr\" (UniqueName: \"kubernetes.io/projected/5bea7640-b694-47b4-8881-28ae3c025d10-kube-api-access-kxnmr\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.788826 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.788738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-metrics-certs\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.789247 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.789221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e652e62b-617d-42e8-9863-659445f40674-openshift-service-ca\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.789457 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.789306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea7640-b694-47b4-8881-28ae3c025d10-openshift-service-ca\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.791181 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.791152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e652e62b-617d-42e8-9863-659445f40674-maas-api-tls\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.791312 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.791254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-metrics-certs\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.791396 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.791363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-maas-api-tls\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.791396 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.791367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e652e62b-617d-42e8-9863-659445f40674-metrics-certs\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.796706 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.796683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxnmr\" (UniqueName: \"kubernetes.io/projected/5bea7640-b694-47b4-8881-28ae3c025d10-kube-api-access-kxnmr\") pod \"maas-api-577f88484f-pn8mr\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.796789 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.796687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2kw\" (UniqueName: \"kubernetes.io/projected/e652e62b-617d-42e8-9863-659445f40674-kube-api-access-gz2kw\") pod \"maas-api-778c967957-c5rrn\" (UID: \"e652e62b-617d-42e8-9863-659445f40674\") " pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:47.866528 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.866494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:47.881962 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:47.881933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:48.142719 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:48.142684    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"]
Sep 11 00:13:48.144644 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:13:48.144614    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bea7640_b694_47b4_8881_28ae3c025d10.slice/crio-167870542c47efd87fce16e18533d5087ed35bc602027f29c7674b48785e9288 WatchSource:0}: Error finding container 167870542c47efd87fce16e18533d5087ed35bc602027f29c7674b48785e9288: Status 404 returned error can't find the container with id 167870542c47efd87fce16e18533d5087ed35bc602027f29c7674b48785e9288
Sep 11 00:13:48.146124 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:48.146108    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 00:13:48.169431 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:48.169402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-778c967957-c5rrn"]
Sep 11 00:13:48.172909 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:13:48.172882    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode652e62b_617d_42e8_9863_659445f40674.slice/crio-5e096858292487bafd2b687428388ea3f2db371b79c22107393c9cf7613dc342 WatchSource:0}: Error finding container 5e096858292487bafd2b687428388ea3f2db371b79c22107393c9cf7613dc342: Status 404 returned error can't find the container with id 5e096858292487bafd2b687428388ea3f2db371b79c22107393c9cf7613dc342
Sep 11 00:13:48.560829 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:48.560788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn" event={"ID":"e652e62b-617d-42e8-9863-659445f40674","Type":"ContainerStarted","Data":"5e096858292487bafd2b687428388ea3f2db371b79c22107393c9cf7613dc342"}
Sep 11 00:13:48.561914 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:48.561891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr" event={"ID":"5bea7640-b694-47b4-8881-28ae3c025d10","Type":"ContainerStarted","Data":"167870542c47efd87fce16e18533d5087ed35bc602027f29c7674b48785e9288"}
Sep 11 00:13:49.568405 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:49.568367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn" event={"ID":"e652e62b-617d-42e8-9863-659445f40674","Type":"ContainerStarted","Data":"324d6ddf983f2c399bb03ed39a5aad2ae594fc3865b52d83a95bf311c39c7697"}
Sep 11 00:13:49.568911 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:49.568535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:49.569945 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:49.569919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr" event={"ID":"5bea7640-b694-47b4-8881-28ae3c025d10","Type":"ContainerStarted","Data":"fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4"}
Sep 11 00:13:49.570055 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:49.569998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:49.584458 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:49.584416    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn" podStartSLOduration=2.157109499 podStartE2EDuration="2.584401021s" totalImagesPullingTime="427.291522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:13:49.583935431 +0000 UTC m=+2271.540167234" watchObservedRunningTime="2026-09-11 00:13:49.584401021 +0000 UTC m=+2271.540632823"
Sep 11 00:13:49.600107 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:49.600071    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr" podStartSLOduration=2.122875392 podStartE2EDuration="2.600055667s" totalImagesPullingTime="477.180275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:13:49.599070634 +0000 UTC m=+2271.555302437" watchObservedRunningTime="2026-09-11 00:13:49.600055667 +0000 UTC m=+2271.556287470"
Sep 11 00:13:54.889899 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:54.889863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:13:55.579339 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.579309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:55.579804 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.579783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-778c967957-c5rrn"
Sep 11 00:13:55.640922 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.640886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"]
Sep 11 00:13:55.641141 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.641116    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr" podUID="5bea7640-b694-47b4-8881-28ae3c025d10" containerName="maas-api" containerID="cri-o://fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4" gracePeriod=30
Sep 11 00:13:55.886549 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.886525    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:55.958901 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.958868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxnmr\" (UniqueName: \"kubernetes.io/projected/5bea7640-b694-47b4-8881-28ae3c025d10-kube-api-access-kxnmr\") pod \"5bea7640-b694-47b4-8881-28ae3c025d10\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") "
Sep 11 00:13:55.959374 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.958912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea7640-b694-47b4-8881-28ae3c025d10-openshift-service-ca\") pod \"5bea7640-b694-47b4-8881-28ae3c025d10\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") "
Sep 11 00:13:55.959374 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.958941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-maas-api-tls\") pod \"5bea7640-b694-47b4-8881-28ae3c025d10\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") "
Sep 11 00:13:55.959374 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.958961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-metrics-certs\") pod \"5bea7640-b694-47b4-8881-28ae3c025d10\" (UID: \"5bea7640-b694-47b4-8881-28ae3c025d10\") "
Sep 11 00:13:55.959374 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.959303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bea7640-b694-47b4-8881-28ae3c025d10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bea7640-b694-47b4-8881-28ae3c025d10" (UID: "5bea7640-b694-47b4-8881-28ae3c025d10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:13:55.960981 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.960948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bea7640-b694-47b4-8881-28ae3c025d10-kube-api-access-kxnmr" (OuterVolumeSpecName: "kube-api-access-kxnmr") pod "5bea7640-b694-47b4-8881-28ae3c025d10" (UID: "5bea7640-b694-47b4-8881-28ae3c025d10"). InnerVolumeSpecName "kube-api-access-kxnmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:13:55.961093 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.961049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5bea7640-b694-47b4-8881-28ae3c025d10" (UID: "5bea7640-b694-47b4-8881-28ae3c025d10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:13:55.961093 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:55.961080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bea7640-b694-47b4-8881-28ae3c025d10" (UID: "5bea7640-b694-47b4-8881-28ae3c025d10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:13:56.060018 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.059989    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-maas-api-tls\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:56.060018 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.060015    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea7640-b694-47b4-8881-28ae3c025d10-metrics-certs\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:56.060018 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.060024    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxnmr\" (UniqueName: \"kubernetes.io/projected/5bea7640-b694-47b4-8881-28ae3c025d10-kube-api-access-kxnmr\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:56.060222 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.060033    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea7640-b694-47b4-8881-28ae3c025d10-openshift-service-ca\") on node \"ip-10-0-128-88.ec2.internal\" DevicePath \"\""
Sep 11 00:13:56.600373 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.600341    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bea7640-b694-47b4-8881-28ae3c025d10" containerID="fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4" exitCode=0
Sep 11 00:13:56.600542 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.600389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr" event={"ID":"5bea7640-b694-47b4-8881-28ae3c025d10","Type":"ContainerDied","Data":"fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4"}
Sep 11 00:13:56.600542 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.600394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"
Sep 11 00:13:56.600542 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.600429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-pn8mr" event={"ID":"5bea7640-b694-47b4-8881-28ae3c025d10","Type":"ContainerDied","Data":"167870542c47efd87fce16e18533d5087ed35bc602027f29c7674b48785e9288"}
Sep 11 00:13:56.600542 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.600450    2578 scope.go:117] "RemoveContainer" containerID="fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4"
Sep 11 00:13:56.608915 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.608884    2578 scope.go:117] "RemoveContainer" containerID="fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4"
Sep 11 00:13:56.609142 ip-10-0-128-88 kubenswrapper[2578]: E0911 00:13:56.609121    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4\": container with ID starting with fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4 not found: ID does not exist" containerID="fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4"
Sep 11 00:13:56.609243 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.609147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4"} err="failed to get container status \"fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4\": rpc error: code = NotFound desc = could not find container \"fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4\": container with ID starting with fa9895d80e82da57c9bf46e4a39c9405009c46814e7c7c1de6cef47bb979aca4 not found: ID does not exist"
Sep 11 00:13:56.616621 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.616598    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"]
Sep 11 00:13:56.621490 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:56.621469    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-pn8mr"]
Sep 11 00:13:58.536976 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:13:58.536945    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bea7640-b694-47b4-8881-28ae3c025d10" path="/var/lib/kubelet/pods/5bea7640-b694-47b4-8881-28ae3c025d10/volumes"
Sep 11 00:14:02.088181 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:02.088148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:14:07.888439 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:07.888399    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:14:14.291814 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:14.291777    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bkvpx"]
Sep 11 00:14:20.123408 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:20.123368    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:14:23.212032 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:23.212002    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-ktb4q_99ec7f0c-09a7-4f3a-8331-6812961f6256/copy-manifests/0.log"
Sep 11 00:14:23.225562 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:23.225534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-ktb4q_99ec7f0c-09a7-4f3a-8331-6812961f6256/manager/0.log"
Sep 11 00:14:23.666507 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:23.666475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-778c967957-c5rrn_e652e62b-617d-42e8-9863-659445f40674/maas-api/0.log"
Sep 11 00:14:25.232121 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:25.232087    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-s69jt_98e883ef-8d64-4061-830a-f068947f788f/manager/0.log"
Sep 11 00:14:25.347525 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:25.347500    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-bkvpx_02bda458-d06e-4447-be8c-549cc03058b0/limitador/0.log"
Sep 11 00:14:26.017613 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:26.017578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-547c6fc866-zjtsj_0bb639d0-026a-4a2e-a501-2a7e921288f5/kube-auth-proxy/0.log"
Sep 11 00:14:26.579745 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:26.579707    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84c49bb84b-8nk86_4cfb42b9-bb91-4012-879e-b185d1cce2eb/router/0.log"
Sep 11 00:14:27.229409 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:27.229373    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2cncz_3faac05e-1552-4184-8c2e-4b3f5eb44340/main/0.log"
Sep 11 00:14:27.604107 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:27.604075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-f582d_a21186bb-54f4-4c9a-baeb-30e0e21ea30d/main/0.log"
Sep 11 00:14:32.454392 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.454356    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-chw9w/must-gather-tqm7b"]
Sep 11 00:14:32.454882 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.454862    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bea7640-b694-47b4-8881-28ae3c025d10" containerName="maas-api"
Sep 11 00:14:32.454972 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.454887    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bea7640-b694-47b4-8881-28ae3c025d10" containerName="maas-api"
Sep 11 00:14:32.455030 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.455004    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bea7640-b694-47b4-8881-28ae3c025d10" containerName="maas-api"
Sep 11 00:14:32.457970 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.457950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:32.460575 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.460543    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-chw9w\"/\"openshift-service-ca.crt\""
Sep 11 00:14:32.461733 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.461714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-chw9w\"/\"kube-root-ca.crt\""
Sep 11 00:14:32.461807 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.461775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-chw9w\"/\"default-dockercfg-w5c67\""
Sep 11 00:14:32.466448 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.466428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chw9w/must-gather-tqm7b"]
Sep 11 00:14:32.471875 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.471849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:14:32.570022 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.569994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log" limit=1024
Sep 11 00:14:32.587570 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.587546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9dd1c44f-f32f-42f6-9a52-9a4b2cefa916-must-gather-output\") pod \"must-gather-tqm7b\" (UID: \"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916\") " pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:32.587642 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.587588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzb4m\" (UniqueName: \"kubernetes.io/projected/9dd1c44f-f32f-42f6-9a52-9a4b2cefa916-kube-api-access-dzb4m\") pod \"must-gather-tqm7b\" (UID: \"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916\") " pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:32.688254 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.688223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9dd1c44f-f32f-42f6-9a52-9a4b2cefa916-must-gather-output\") pod \"must-gather-tqm7b\" (UID: \"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916\") " pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:32.688402 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.688261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzb4m\" (UniqueName: \"kubernetes.io/projected/9dd1c44f-f32f-42f6-9a52-9a4b2cefa916-kube-api-access-dzb4m\") pod \"must-gather-tqm7b\" (UID: \"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916\") " pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:32.688551 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.688531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9dd1c44f-f32f-42f6-9a52-9a4b2cefa916-must-gather-output\") pod \"must-gather-tqm7b\" (UID: \"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916\") " pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:32.696977 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.696952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzb4m\" (UniqueName: \"kubernetes.io/projected/9dd1c44f-f32f-42f6-9a52-9a4b2cefa916-kube-api-access-dzb4m\") pod \"must-gather-tqm7b\" (UID: \"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916\") " pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:32.767962 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:32.767942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chw9w/must-gather-tqm7b"
Sep 11 00:14:33.036670 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:33.036580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chw9w/must-gather-tqm7b"]
Sep 11 00:14:33.039009 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:14:33.038985    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dd1c44f_f32f_42f6_9a52_9a4b2cefa916.slice/crio-51420b8aab5b824e2ac56dd7cd110ef7cd09f0582fc60f040c9599247b51506b WatchSource:0}: Error finding container 51420b8aab5b824e2ac56dd7cd110ef7cd09f0582fc60f040c9599247b51506b: Status 404 returned error can't find the container with id 51420b8aab5b824e2ac56dd7cd110ef7cd09f0582fc60f040c9599247b51506b
Sep 11 00:14:33.770623 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:33.770593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chw9w/must-gather-tqm7b" event={"ID":"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916","Type":"ContainerStarted","Data":"51420b8aab5b824e2ac56dd7cd110ef7cd09f0582fc60f040c9599247b51506b"}
Sep 11 00:14:34.779624 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:34.779530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chw9w/must-gather-tqm7b" event={"ID":"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916","Type":"ContainerStarted","Data":"edc9efd3b1fd508382ea9ed5a198979268327f5577c0b5f686092e963b64bb63"}
Sep 11 00:14:34.779624 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:34.779582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chw9w/must-gather-tqm7b" event={"ID":"9dd1c44f-f32f-42f6-9a52-9a4b2cefa916","Type":"ContainerStarted","Data":"2b1881ec058f67cc2d8c9382e77ba3764154392f668e738bdf160138df1cfe54"}
Sep 11 00:14:34.796133 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:34.796073    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-chw9w/must-gather-tqm7b" podStartSLOduration=2.152332066 podStartE2EDuration="2.796055362s" totalImagesPullingTime="643.723296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:14:34.795484696 +0000 UTC m=+2316.751716501" watchObservedRunningTime="2026-09-11 00:14:34.796055362 +0000 UTC m=+2316.752287164"
Sep 11 00:14:35.180858 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:35.180781    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5lhn7_5b828bf5-24ab-4b4c-8d09-2279c1c4c709/global-pull-secret-syncer/0.log"
Sep 11 00:14:35.313828 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:35.313787    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6bkhx_a18d1704-9495-49e1-912d-b1da73e3da26/konnectivity-agent/0.log"
Sep 11 00:14:35.404350 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:35.404319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-88.ec2.internal_0c87a324f1802bb8036fb2caa110222f/haproxy/0.log"
Sep 11 00:14:40.169421 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:40.169382    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-s69jt_98e883ef-8d64-4061-830a-f068947f788f/manager/0.log"
Sep 11 00:14:40.199626 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:40.199577    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-bkvpx_02bda458-d06e-4447-be8c-549cc03058b0/limitador/0.log"
Sep 11 00:14:41.910868 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:41.910830    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-j9fhz_cb87bdeb-36f6-4a0e-a3a8-e5f436232321/kube-state-metrics/0.log"
Sep 11 00:14:41.939673 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:41.939589    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-j9fhz_cb87bdeb-36f6-4a0e-a3a8-e5f436232321/kube-rbac-proxy-main/0.log"
Sep 11 00:14:41.958527 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:41.958495    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-j9fhz_cb87bdeb-36f6-4a0e-a3a8-e5f436232321/kube-rbac-proxy-self/0.log"
Sep 11 00:14:41.983648 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:41.983611    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78846476b6-cbm59_79088aca-c73e-452a-9002-2e953505018a/metrics-server/0.log"
Sep 11 00:14:42.014999 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.014970    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-5pf98_3a01c458-2e83-4321-9ba8-5da89b87ef3c/monitoring-plugin/0.log"
Sep 11 00:14:42.114329 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.114298    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpb6j_3be8851d-6eba-4ac5-9ba1-13a13caeec1d/node-exporter/0.log"
Sep 11 00:14:42.133103 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.133075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpb6j_3be8851d-6eba-4ac5-9ba1-13a13caeec1d/kube-rbac-proxy/0.log"
Sep 11 00:14:42.152951 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.152917    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpb6j_3be8851d-6eba-4ac5-9ba1-13a13caeec1d/init-textfile/0.log"
Sep 11 00:14:42.250629 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.250511    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dl42r_1faa6ed1-d13b-4b81-aeaf-125eb3ccc358/kube-rbac-proxy-main/0.log"
Sep 11 00:14:42.269614 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.269575    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dl42r_1faa6ed1-d13b-4b81-aeaf-125eb3ccc358/kube-rbac-proxy-self/0.log"
Sep 11 00:14:42.290792 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.290759    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dl42r_1faa6ed1-d13b-4b81-aeaf-125eb3ccc358/openshift-state-metrics/0.log"
Sep 11 00:14:42.326588 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.326552    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e1d0a207-3f3f-40b0-8e61-7470d99444ee/prometheus/0.log"
Sep 11 00:14:42.348012 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.347976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e1d0a207-3f3f-40b0-8e61-7470d99444ee/config-reloader/0.log"
Sep 11 00:14:42.372806 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.372770    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e1d0a207-3f3f-40b0-8e61-7470d99444ee/thanos-sidecar/0.log"
Sep 11 00:14:42.394846 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.394800    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e1d0a207-3f3f-40b0-8e61-7470d99444ee/kube-rbac-proxy-web/0.log"
Sep 11 00:14:42.414635 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.414604    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e1d0a207-3f3f-40b0-8e61-7470d99444ee/kube-rbac-proxy/0.log"
Sep 11 00:14:42.436091 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.436059    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e1d0a207-3f3f-40b0-8e61-7470d99444ee/kube-rbac-proxy-thanos/0.log"
Sep 11 00:14:42.459265 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.459227    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e1d0a207-3f3f-40b0-8e61-7470d99444ee/init-config-reloader/0.log"
Sep 11 00:14:42.492621 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.492587    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wnhww_1ac4929b-70d2-4596-a915-a74186b90320/prometheus-operator/0.log"
Sep 11 00:14:42.523323 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.523268    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wnhww_1ac4929b-70d2-4596-a915-a74186b90320/kube-rbac-proxy/0.log"
Sep 11 00:14:42.580292 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:42.580243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-qsf5b_5da565e0-536c-416a-9366-98d8ce4e62db/prometheus-operator-admission-webhook/0.log"
Sep 11 00:14:45.057041 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.057009    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/2.log"
Sep 11 00:14:45.067150 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.067122    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sq4f8_2a008749-a748-49c8-abc4-a0506f8f2e69/console-operator/3.log"
Sep 11 00:14:45.637103 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.637065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-jnpnm_25f6aa54-56b3-454a-81b6-fe8a56a31fb3/download-server/0.log"
Sep 11 00:14:45.691765 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.691729    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"]
Sep 11 00:14:45.701367 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.701340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"]
Sep 11 00:14:45.701697 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.701641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.718310 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.718248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-proc\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.718461 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.718342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-sys\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.718461 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.718370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-podres\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.718596 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.718460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7rt\" (UniqueName: \"kubernetes.io/projected/123d4588-d5bb-4e1a-b27b-8cfc38499882-kube-api-access-6n7rt\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.718596 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.718532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-lib-modules\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819102 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-lib-modules\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819102 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-proc\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819376 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-sys\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819376 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-podres\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819376 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7rt\" (UniqueName: \"kubernetes.io/projected/123d4588-d5bb-4e1a-b27b-8cfc38499882-kube-api-access-6n7rt\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819776 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-sys\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819776 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-proc\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819776 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-podres\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.819776 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.819741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/123d4588-d5bb-4e1a-b27b-8cfc38499882-lib-modules\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:45.827818 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:45.827754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7rt\" (UniqueName: \"kubernetes.io/projected/123d4588-d5bb-4e1a-b27b-8cfc38499882-kube-api-access-6n7rt\") pod \"perf-node-gather-daemonset-jlqf9\" (UID: \"123d4588-d5bb-4e1a-b27b-8cfc38499882\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:46.017256 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:46.017207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:46.317134 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:46.317108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"]
Sep 11 00:14:46.327021 ip-10-0-128-88 kubenswrapper[2578]: W0911 00:14:46.326986    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod123d4588_d5bb_4e1a_b27b_8cfc38499882.slice/crio-83c9611be91e1dd86599cb4e9a3e9d4083dd082883ec172e6627e08777ff6863 WatchSource:0}: Error finding container 83c9611be91e1dd86599cb4e9a3e9d4083dd082883ec172e6627e08777ff6863: Status 404 returned error can't find the container with id 83c9611be91e1dd86599cb4e9a3e9d4083dd082883ec172e6627e08777ff6863
Sep 11 00:14:46.863254 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:46.863208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9" event={"ID":"123d4588-d5bb-4e1a-b27b-8cfc38499882","Type":"ContainerStarted","Data":"f097db067d7b3bfcaeabf5ff6de5ef4b68509e1f34024fcf28225e43a4e7387a"}
Sep 11 00:14:46.863464 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:46.863289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9" event={"ID":"123d4588-d5bb-4e1a-b27b-8cfc38499882","Type":"ContainerStarted","Data":"83c9611be91e1dd86599cb4e9a3e9d4083dd082883ec172e6627e08777ff6863"}
Sep 11 00:14:46.863778 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:46.863758    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:46.884915 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:46.884736    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9" podStartSLOduration=1.884716347 podStartE2EDuration="1.884716347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:14:46.881195717 +0000 UTC m=+2328.837427536" watchObservedRunningTime="2026-09-11 00:14:46.884716347 +0000 UTC m=+2328.840948151"
Sep 11 00:14:47.502428 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:47.502397    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bzgp5_f1c2427c-a1e9-4e53-a2de-e7b8630507fe/dns/0.log"
Sep 11 00:14:47.522605 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:47.522564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bzgp5_f1c2427c-a1e9-4e53-a2de-e7b8630507fe/kube-rbac-proxy/0.log"
Sep 11 00:14:47.632832 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:47.632796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bfnfj_8ae7c2aa-ebd2-4925-a132-c5b91088a016/dns-node-resolver/0.log"
Sep 11 00:14:48.349286 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:48.349243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmbws_453364b7-bff4-4640-8fb2-07ba8b5f95c9/node-ca/0.log"
Sep 11 00:14:49.827324 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:49.827268    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-547c6fc866-zjtsj_0bb639d0-026a-4a2e-a501-2a7e921288f5/kube-auth-proxy/0.log"
Sep 11 00:14:49.972451 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:49.972415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84c49bb84b-8nk86_4cfb42b9-bb91-4012-879e-b185d1cce2eb/router/0.log"
Sep 11 00:14:50.843572 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:50.843535    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v5jdk_880f83bd-a861-4ca0-9e53-b84389e50e8d/serve-healthcheck-canary/0.log"
Sep 11 00:14:51.392487 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:51.392456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hrvpr_36fe9cad-01bc-41f0-8718-85cd9b5ba32d/kube-rbac-proxy/0.log"
Sep 11 00:14:51.410899 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:51.410870    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hrvpr_36fe9cad-01bc-41f0-8718-85cd9b5ba32d/exporter/0.log"
Sep 11 00:14:51.430680 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:51.430645    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hrvpr_36fe9cad-01bc-41f0-8718-85cd9b5ba32d/extractor/0.log"
Sep 11 00:14:53.885101 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:53.885068    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-jlqf9"
Sep 11 00:14:53.968287 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:53.968236    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-ktb4q_99ec7f0c-09a7-4f3a-8331-6812961f6256/manager/0.log"
Sep 11 00:14:53.986771 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:53.986738    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-ktb4q_99ec7f0c-09a7-4f3a-8331-6812961f6256/copy-manifests/0.log"
Sep 11 00:14:55.189789 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:55.189716    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-98d67b8cf-hrlhr_777661a7-5930-4dbd-a1d2-6db42f470a23/manager/0.log"
Sep 11 00:14:56.346204 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:56.346171    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-547c6fc866-zjtsj_0bb639d0-026a-4a2e-a501-2a7e921288f5/kube-auth-proxy/0.log"
Sep 11 00:14:56.458871 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:56.458796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84c49bb84b-8nk86_4cfb42b9-bb91-4012-879e-b185d1cce2eb/router/0.log"
Sep 11 00:14:58.639345 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:58.639255    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-547c6fc866-zjtsj_0bb639d0-026a-4a2e-a501-2a7e921288f5/kube-auth-proxy/0.log"
Sep 11 00:14:58.759611 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:14:58.759537    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84c49bb84b-8nk86_4cfb42b9-bb91-4012-879e-b185d1cce2eb/router/0.log"
Sep 11 00:15:01.859211 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:01.859172    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9fm6t_5236291c-b0c3-480c-bb27-f29ff23f8c42/kube-multus/0.log"
Sep 11 00:15:02.025253 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.025221    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk2k7_4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d/kube-multus-additional-cni-plugins/0.log"
Sep 11 00:15:02.046221 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.046188    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk2k7_4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d/egress-router-binary-copy/0.log"
Sep 11 00:15:02.064061 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.064035    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk2k7_4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d/cni-plugins/0.log"
Sep 11 00:15:02.083293 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.083238    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk2k7_4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d/bond-cni-plugin/0.log"
Sep 11 00:15:02.101844 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.101814    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk2k7_4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d/routeoverride-cni/0.log"
Sep 11 00:15:02.119164 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.119096    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk2k7_4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d/whereabouts-cni-bincopy/0.log"
Sep 11 00:15:02.137216 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.137196    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk2k7_4cdd04d5-18fa-41e3-a658-1fe35ee0ad8d/whereabouts-cni/0.log"
Sep 11 00:15:02.466465 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.466401    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z6rwl_4a62a794-f5a8-423e-bb4d-20ec9edccf4b/network-metrics-daemon/0.log"
Sep 11 00:15:02.479758 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:02.479730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z6rwl_4a62a794-f5a8-423e-bb4d-20ec9edccf4b/kube-rbac-proxy/0.log"
Sep 11 00:15:03.467888 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.467857    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/ovn-controller/0.log"
Sep 11 00:15:03.481516 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.481490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/ovn-acl-logging/0.log"
Sep 11 00:15:03.506934 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.506908    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/kube-rbac-proxy-node/0.log"
Sep 11 00:15:03.526088 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.526062    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 00:15:03.540635 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.540613    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/northd/0.log"
Sep 11 00:15:03.558826 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.558798    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/nbdb/0.log"
Sep 11 00:15:03.578010 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.577984    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/sbdb/0.log"
Sep 11 00:15:03.679536 ip-10-0-128-88 kubenswrapper[2578]: I0911 00:15:03.679456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8rhr_7dba2314-a579-4782-8272-e1af9b28a637/ovnkube-controller/0.log"