Sep 16 00:43:01.346800 ip-10-0-130-4 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 00:43:01.728582 ip-10-0-130-4 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:43:01.728582 ip-10-0-130-4 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 00:43:01.728582 ip-10-0-130-4 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:43:01.728582 ip-10-0-130-4 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 00:43:01.728582 ip-10-0-130-4 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:43:01.729251 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.728652    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 00:43:01.731468 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731451    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:01.731468 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731467    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731471    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731474    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731477    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731495    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731500    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731503    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731506    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731509    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731512    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731515    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731518    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731520    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731530    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731533    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731536    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731539    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731541    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731544    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731547    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:01.731547 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731550    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731553    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731556    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731559    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731562    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731564    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731567    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731569    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731572    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731574    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731577    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731580    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731582    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731585    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731587    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731590    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731593    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731595    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731598    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:01.732016 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731603    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731607    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731610    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731613    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731616    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731619    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731621    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731625    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731627    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731630    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731633    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731635    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731638    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731640    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731643    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731646    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731649    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731651    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731654    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:01.732541 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731658    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731661    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731664    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731666    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731669    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731671    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731674    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731678    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731681    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731684    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731687    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731690    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731692    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731695    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731698    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731700    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731703    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731706    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731708    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731711    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:01.733002 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731714    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731718    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731720    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731723    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731726    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731728    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731731    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731735    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731738    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731741    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731743    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731746    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731748    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731751    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731754    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731756    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731758    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731762    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731766    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.731770    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:01.733488 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732450    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732456    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732459    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732462    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732465    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732467    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732470    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732474    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732477    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732492    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732495    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732497    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732500    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732503    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732506    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732509    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732511    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732514    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732517    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:01.733974 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732520    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732525    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732530    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732534    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732537    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732540    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732543    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732546    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732549    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732552    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732555    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732557    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732560    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732563    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732565    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732568    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732571    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732573    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732577    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732579    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:01.734440 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732582    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732585    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732587    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732590    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732592    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732595    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732597    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732600    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732603    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732606    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732608    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732611    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732614    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732617    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732621    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732625    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732628    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732631    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732633    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:01.734944 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732636    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732638    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732641    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732644    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732646    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732649    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732651    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732654    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732656    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732659    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732661    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732664    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732666    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732669    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732671    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732674    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732676    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732679    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732681    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732684    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:01.735406 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732686    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732689    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732691    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732694    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732699    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732701    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732704    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732708    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732712    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732715    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732718    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732721    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732724    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732727    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732730    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732732    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732735    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732737    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732740    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:01.735982 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732743    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.732745    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733140    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733145    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733148    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733151    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733154    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733157    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733160    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733163    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733166    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733168    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733171    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733173    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733176    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733179    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733183    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733187    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733189    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733193    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:01.736493 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733196    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733199    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733202    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733204    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733207    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733210    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733213    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733215    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733218    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733221    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733224    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733226    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733229    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733231    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733234    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733236    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733238    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733241    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733244    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733246    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:01.736997 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733248    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733251    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733253    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733256    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733258    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733260    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733263    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733265    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733268    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733271    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733273    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733276    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733279    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733281    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733284    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733287    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733289    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733292    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733294    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733297    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:01.737476 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733299    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733302    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733305    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733307    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733310    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733312    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733315    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733317    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733320    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733322    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733324    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733327    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733329    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733332    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733335    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733338    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733341    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733344    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733347    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:01.738005 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733349    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733352    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733355    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733358    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733361    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733364    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733366    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733369    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733372    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733375    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733377    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733380    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733382    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733385    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733388    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733390    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733392    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733395    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733398    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:01.738474 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733402    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733405    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.733408    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733477    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733530    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733537    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733542    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733546    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733550    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733554    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733558    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733561    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733565    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733568    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733572    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733577    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733581    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733584    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733587    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733590    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733593    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733598    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733601    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733604    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 00:43:01.738951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733607    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733610    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733614    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733617    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733620    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733623    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733626    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733629    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733632    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733636    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733639    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733643    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733646    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733653    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733656    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733659    2581 flags.go:64] FLAG: --enable-server="true"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733662    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733667    2581 flags.go:64] FLAG: --event-burst="100"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733670    2581 flags.go:64] FLAG: --event-qps="50"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733672    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733675    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733679    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733682    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733686    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733689    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 00:43:01.739581 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733694    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733697    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733700    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733703    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733706    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733709    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733712    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733714    2581 flags.go:64] FLAG: --feature-gates=""
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733718    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733721    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733724    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733727    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733730    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733733    2581 flags.go:64] FLAG: --help="false"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733736    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-4.ec2.internal"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733739    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733742    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733745    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733748    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733752    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733756    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733758    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733762    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 00:43:01.740188 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733765    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733768    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733771    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733774    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733777    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733779    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733782    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733786    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733789    2581 flags.go:64] FLAG: --lock-file=""
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733792    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733796    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733799    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733804    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733807    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733810    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733813    2581 flags.go:64] FLAG: --logging-format="text"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733816    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733819    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733822    2581 flags.go:64] FLAG: --manifest-url=""
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733825    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733829    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733832    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733836    2581 flags.go:64] FLAG: --max-pods="110"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733839    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733842    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 00:43:01.740867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733845    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733848    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733851    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733854    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733859    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733865    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733868    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733872    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733875    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733878    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733883    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733885    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733888    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733891    2581 flags.go:64] FLAG: --port="10250"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733896    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733899    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00944113167b45694"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733902    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733905    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733908    2581 flags.go:64] FLAG: --register-node="true"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733912    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733915    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733918    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733921    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733924    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733933    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 00:43:01.741566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733936    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733940    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733943    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733946    2581 flags.go:64] FLAG: --runonce="false"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733949    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733952    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733956    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733959    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733963    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733966    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733969    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733973    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733977    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733981    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733984    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733988    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733991    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733994    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.733997    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734002    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734006    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734010    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734015    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734018    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734021    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 00:43:01.742236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734024    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734027    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734031    2581 flags.go:64] FLAG: --v="2"
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734035    2581 flags.go:64] FLAG: --version="false"
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734039    2581 flags.go:64] FLAG: --vmodule=""
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734043    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.734046    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734160    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734164    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734167    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734170    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734173    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734176    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734178    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734181    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734184    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734186    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734189    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734192    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734196    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734198    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:01.742927 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734202    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734204    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734207    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734209    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734212    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734215    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734218    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734222    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734224    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734227    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734229    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734232    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734234    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734240    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734244    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734247    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734250    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734253    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734256    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:01.743545 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734258    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734261    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734263    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734266    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734269    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734271    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734274    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734276    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734279    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734281    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734284    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734289    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734293    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734296    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734298    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734301    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734304    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734307    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734309    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734312    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:01.744058 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734316    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734318    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734321    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734324    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734326    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734329    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734331    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734335    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734337    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734340    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734343    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734345    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734348    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734350    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734353    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734355    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734358    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734361    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734363    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734366    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:01.744589 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734368    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734371    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734373    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734377    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734381    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734385    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734388    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734392    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734396    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734399    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734402    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734405    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734409    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734411    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734415    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734419    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734421    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734424    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734427    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734429    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:01.745100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734432    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734435    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734438    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734440    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734443    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.734446    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.735317    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.741959    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.741976    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.743259    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.743271    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743329    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743334    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743338    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743341    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:01.745614 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743344    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743347    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743350    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743354    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743358    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743361    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743364    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743367    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743369    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743373    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743375    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743378    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743382    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743386    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743389    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743392    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743394    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743397    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743400    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:01.745991 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743403    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743405    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743408    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743410    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743413    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743416    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743420    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743423    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743425    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743428    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743431    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743434    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743436    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743439    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743441    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743444    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743447    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743449    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743452    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743454    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743457    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:01.746460 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743459    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743462    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743465    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743467    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743470    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743472    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743475    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743502    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743506    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743511    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743515    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743518    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743521    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743524    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743528    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743531    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743534    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743537    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743540    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:01.747032 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743543    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743546    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743549    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743551    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743555    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743558    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743560    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743563    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743566    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743568    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743571    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743574    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743577    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743579    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743582    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743584    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743587    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743589    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743592    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743595    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:01.747519 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743598    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743600    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743603    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743606    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743609    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743612    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743614    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743617    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743620    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743623    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743626    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743628    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743632    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743634    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743638    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:01.747996 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743640    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.743645    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743780    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743786    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743789    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743792    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743795    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743798    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743801    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743804    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743807    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743810    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743812    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743815    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743818    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:01.748361 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743820    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743824    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743828    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743830    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743833    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743836    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743838    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743841    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743844    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743846    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743849    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743851    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743854    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743857    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743860    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743863    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743865    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743868    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743871    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:01.748751 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743873    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743876    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743879    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743881    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743884    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743886    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743889    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743891    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743894    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743896    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743899    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743901    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743904    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743907    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743910    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743912    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743915    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743917    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743920    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:01.749208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743922    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743925    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743928    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743930    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743934    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743938    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743941    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743944    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743948    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743951    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743955    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743958    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743960    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743963    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743965    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743968    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743970    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743973    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743976    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743978    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:01.749694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743980    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743983    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743985    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743988    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743991    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743993    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743996    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.743998    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744001    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744004    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744006    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744009    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744012    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744014    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744017    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744019    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744021    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744024    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744027    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744029    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:01.750190 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744032    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744034    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744037    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744041    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744043    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744046    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744048    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:01.744051    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.744055    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.744168    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.746328    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.747288    2581 server.go:1019] "Starting client certificate rotation"
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.747386    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 00:43:01.750692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.748168    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 00:43:01.751007 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.750429    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 00:43:01.752137 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.752117    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 00:43:01.766544 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.766519    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 16 00:43:01.769660 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.769632    2581 log.go:25] "Validated CRI v1 image API"
Sep 16 00:43:01.770506 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.770471    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 00:43:01.773569 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.773544    2581 fs.go:135] Filesystem UUIDs: map[463d6115-214e-4c61-9c95-1f48f32c20b9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a11e2772-0bbb-4543-916a-8e2b84eed3dc:/dev/nvme0n1p4]
Sep 16 00:43:01.773653 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.773567    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 00:43:01.777252 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.777233    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 00:43:01.779843 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.779719    2581 manager.go:216] Machine: {Timestamp:2026-09-16 00:43:01.777528973 +0000 UTC m=+0.320665938 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096651 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec264cf0d5391aa3571f9bbb6f16dc68 SystemUUID:ec264cf0-d539-1aa3-571f-9bbb6f16dc68 BootID:67339af9-12a5-4c9b-8271-d011f28cabe4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:f5:51:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:f5:51:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:52:84:3f:cd:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 00:43:01.779843 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.779839    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 00:43:01.779952 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.779930    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 00:43:01.782925 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.782901    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 00:43:01.783084 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.782928    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 00:43:01.783703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.783692    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 00:43:01.783740 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.783706    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 00:43:01.783740 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.783714    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 00:43:01.784538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.784528    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 00:43:01.785281 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.785271    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 00:43:01.787911 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.787900    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 00:43:01.788020 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788011    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 00:43:01.788050 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788039    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 00:43:01.788050 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788049    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 00:43:01.788100 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788066    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 00:43:01.788100 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788075    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 16 00:43:01.788100 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788085    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 00:43:01.788439 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788419    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 00:43:01.788475 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.788447    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 00:43:01.790681 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.790663    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 00:43:01.792662 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.792648    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 00:43:01.792734 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.792678    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 00:43:01.793938 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.793911    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 00:43:01.793938 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.793942    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 00:43:01.794065 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.793949    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 00:43:01.794065 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.793956    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 00:43:01.794065 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794036    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 00:43:01.794065 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794049    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 00:43:01.794207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794072    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 00:43:01.794207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794080    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 00:43:01.794207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794090    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 00:43:01.794207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794111    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 00:43:01.794207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794136    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 00:43:01.794207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794145    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 00:43:01.794875 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794862    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 00:43:01.794908 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.794877    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 00:43:01.799757 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.799742    2581 server.go:1268] "Started kubelet"
Sep 16 00:43:01.799844 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.799805    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 00:43:01.799944 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.799889    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 00:43:01.800036 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.799965    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 00:43:01.800318 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.800297    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 00:43:01.800699 ip-10-0-130-4 systemd[1]: Started Kubernetes Kubelet.
Sep 16 00:43:01.804164 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.804140    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 00:43:01.804164 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.804148    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 00:43:01.804856 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.804823    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 00:43:01.806174 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.806158    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 00:43:01.807032 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.807005    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 00:43:01.810646 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.810625    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 00:43:01.810767 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.810746    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:01.810894 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.810630    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 00:43:01.811211 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.811196    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 00:43:01.811211 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.811210    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 00:43:01.811364 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.811196    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 00:43:01.811705 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.811547    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 00:43:01.811705 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.811578    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 00:43:01.811834 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.811802    2581 factory.go:55] Registering systemd factory
Sep 16 00:43:01.811834 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.811818    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 16 00:43:01.811995 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.810875    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-4.ec2.internal.18d5a6657ed8f435  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-4.ec2.internal,UID:ip-10-0-130-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-4.ec2.internal,},FirstTimestamp:2026-09-16 00:43:01.799695413 +0000 UTC m=+0.342832379,LastTimestamp:2026-09-16 00:43:01.799695413 +0000 UTC m=+0.342832379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-4.ec2.internal,}"
Sep 16 00:43:01.812079 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.812067    2581 factory.go:153] Registering CRI-O factory
Sep 16 00:43:01.812122 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.812096    2581 factory.go:223] Registration of the crio container factory successfully
Sep 16 00:43:01.812163 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.812151    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 00:43:01.812197 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.812176    2581 factory.go:103] Registering Raw factory
Sep 16 00:43:01.812197 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.812192    2581 manager.go:1196] Started watching for new ooms in manager
Sep 16 00:43:01.812620 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.812603    2581 manager.go:349] Starting recovery of all containers
Sep 16 00:43:01.816553 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.816525    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 00:43:01.816815 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.816792    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 00:43:01.823695 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.823674    2581 manager.go:354] Recovery completed
Sep 16 00:43:01.827851 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.827739    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:01.828153 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.828129    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gls7w"
Sep 16 00:43:01.829669 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.829650    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:01.829751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.829683    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:01.829751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.829693    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:01.830336 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.830319    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 00:43:01.830336 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.830335    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 00:43:01.830431 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.830343    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:01.830431 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.830355    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 00:43:01.831051 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.831031    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:01.831116 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.831065    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:01.831116 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.831082    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:01.831643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.831629    2581 policy_none.go:49] "None policy: Start"
Sep 16 00:43:01.831711 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.831650    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 00:43:01.831711 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.831664    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 00:43:01.831882 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.831801    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-4.ec2.internal.18d5a66580a24677  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-4.ec2.internal,UID:ip-10-0-130-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-4.ec2.internal,},FirstTimestamp:2026-09-16 00:43:01.829666423 +0000 UTC m=+0.372803396,LastTimestamp:2026-09-16 00:43:01.829666423 +0000 UTC m=+0.372803396,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-4.ec2.internal,}"
Sep 16 00:43:01.832737 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.832724    2581 policy_none.go:47] "Start"
Sep 16 00:43:01.835125 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.835108    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gls7w"
Sep 16 00:43:01.859208 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.859191    2581 manager.go:337] "Starting Device Plugin manager"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.859238    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.859252    2581 server.go:87] "Starting device plugin registration server"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.859606    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.859617    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.859725    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.859808    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.859816    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.860382    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 00:43:01.872835 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.860423    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:01.933059 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.933015    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 00:43:01.934337 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.934321    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 00:43:01.934449 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.934384    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 00:43:01.934449 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.934418    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 00:43:01.934599 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.934526    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 00:43:01.936940 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.936918    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:01.959727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.959699    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:01.960763 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.960742    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:01.960855 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.960775    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:01.960855 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.960790    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:01.960855 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.960823    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-4.ec2.internal"
Sep 16 00:43:01.970983 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:01.970965    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-4.ec2.internal"
Sep 16 00:43:01.971079 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.970992    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-4.ec2.internal\": node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:01.995681 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:01.995605    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.035088 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.035052    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"]
Sep 16 00:43:02.035224 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.035134    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:02.036166 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.036150    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:02.036261 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.036185    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:02.036261 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.036199    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:02.037571 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.037555    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:02.037722 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.037708    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.037775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.037737    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:02.038288 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.038272    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:02.038360 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.038273    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:02.038360 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.038303    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:02.038360 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.038317    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:02.038360 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.038319    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:02.038360 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.038338    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:02.039569 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.039551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.039663 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.039583    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:02.040209 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.040193    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:02.040285 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.040224    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:02.040285 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.040237    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:02.057051 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.057026    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-4.ec2.internal\" not found" node="ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.061056 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.061042    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-4.ec2.internal\" not found" node="ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.096724 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.096681    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.113590 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.113564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b215942334fadb1e5657317484eaae0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal\" (UID: \"2b215942334fadb1e5657317484eaae0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.113728 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.113594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b215942334fadb1e5657317484eaae0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal\" (UID: \"2b215942334fadb1e5657317484eaae0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.113728 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.113612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3855482ecf545e291be557b4f332fc2e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-4.ec2.internal\" (UID: \"3855482ecf545e291be557b4f332fc2e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.197754 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.197714    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.214107 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.214078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b215942334fadb1e5657317484eaae0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal\" (UID: \"2b215942334fadb1e5657317484eaae0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.214203 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.214117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b215942334fadb1e5657317484eaae0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal\" (UID: \"2b215942334fadb1e5657317484eaae0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.214203 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.214141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3855482ecf545e291be557b4f332fc2e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-4.ec2.internal\" (UID: \"3855482ecf545e291be557b4f332fc2e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.214299 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.214193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b215942334fadb1e5657317484eaae0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal\" (UID: \"2b215942334fadb1e5657317484eaae0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.214299 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.214208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b215942334fadb1e5657317484eaae0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal\" (UID: \"2b215942334fadb1e5657317484eaae0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.214299 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.214229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3855482ecf545e291be557b4f332fc2e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-4.ec2.internal\" (UID: \"3855482ecf545e291be557b4f332fc2e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.298570 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.298538    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.362038 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.362001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.364376 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.364355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.399372 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.399330    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.504877 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.504822    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.608947 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.608847    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.710174 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.710133    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-4.ec2.internal\" not found"
Sep 16 00:43:02.737126 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.737100    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:02.746806 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.746776    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 00:43:02.746961 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.746937    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 00:43:02.747003 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.746960    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 00:43:02.790450 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.790418    2581 apiserver.go:52] "Watching apiserver"
Sep 16 00:43:02.803090 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.803066    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:02.810946 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.810914    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 00:43:02.811046 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.811023    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.811126 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.811108    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 00:43:02.813309 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.813284    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pxhn7","kube-system/konnectivity-agent-kwslh","openshift-cluster-node-tuning-operator/tuned-fm86s","openshift-dns/node-resolver-46zrq","openshift-image-registry/node-ca-8z8zz","openshift-multus/multus-additional-cni-plugins-x62zx","openshift-multus/network-metrics-daemon-mftnx","openshift-network-operator/iptables-alerter-vwb6j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z","openshift-multus/multus-v9fk6","openshift-network-diagnostics/network-check-target-h6x8j","openshift-ovn-kubernetes/ovnkube-node-m5btj"]
Sep 16 00:43:02.814774 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.814758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:02.815850 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.815834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:02.817260 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.817080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:02.818268 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.818055    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 00:43:02.818268 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.818054    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 00:43:02.818444 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.818415    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 00:43:02.819224 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.818608    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qn7nn\""
Sep 16 00:43:02.819224 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.818635    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 00:43:02.820380 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.819342    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fvhzp\""
Sep 16 00:43:02.820380 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.819764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.821026 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.820738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r6rxl\""
Sep 16 00:43:02.821026 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.820751    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 00:43:02.821180 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.821094    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 00:43:02.821234 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.821188    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 00:43:02.821514 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.821490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 00:43:02.822145 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.822087    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 00:43:02.822145 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.822109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:02.822145 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.822115    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pztrf\""
Sep 16 00:43:02.822437 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.822173    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:02.822437 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.822111    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 00:43:02.822564 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.822548    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 00:43:02.822616 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.822588    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 00:43:02.822692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.822676    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 00:43:02.823659 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.823638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:02.824471 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.824455    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 00:43:02.824548 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.824532    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"
Sep 16 00:43:02.825063 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.825049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.825681 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.825663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:43:02.825926 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.825907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 00:43:02.826027 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.825937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8sxqn\""
Sep 16 00:43:02.826027 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.825947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 00:43:02.826403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.826387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.827641 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.827445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 00:43:02.827641 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.827486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 00:43:02.827641 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.827498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 00:43:02.827641 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.827525    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q7fk5\""
Sep 16 00:43:02.827919 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.827770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:02.827919 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.827843    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:02.828605 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.828579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 00:43:02.828692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.828664    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rtjm9\""
Sep 16 00:43:02.829273 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.829252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.830726 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.830707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:02.830816 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:02.830759    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:02.831058 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.831041    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 00:43:02.831416 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.831399    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 00:43:02.831582 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.831540    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 00:43:02.831926 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.831910    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal"]
Sep 16 00:43:02.832012 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.831937    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal"]
Sep 16 00:43:02.832087 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.832017    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.832087 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.832042    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 00:43:02.832087 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.832079    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 00:43:02.832279 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.832104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mhfhp\""
Sep 16 00:43:02.832279 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.832135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 00:43:02.832279 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.832242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 00:43:02.834075 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.834060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n77q2\""
Sep 16 00:43:02.834377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.834363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 00:43:02.834565 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.834542    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:43:02.838578 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.838529    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 00:38:02 +0000 UTC" deadline="2028-05-10 19:15:01.915063502 +0000 UTC"
Sep 16 00:43:02.838635 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.838579    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14466h31m59.07648777s"
Sep 16 00:43:02.852962 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.852939    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4g995"
Sep 16 00:43:02.861956 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.861935    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4g995"
Sep 16 00:43:02.912854 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.912829    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 00:43:02.917312 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovnkube-script-lib\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.917413 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysctl-d\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.917413 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hc2\" (UniqueName: \"kubernetes.io/projected/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-kube-api-access-j9hc2\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:02.917413 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-ovn\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.917535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.917535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-socket-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.917535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-etc-kubernetes\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.917535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9bm\" (UniqueName: \"kubernetes.io/projected/30dc45cb-9ef0-411a-88b1-2673963e5977-kube-api-access-dq9bm\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.917692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-run-netns\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.917692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-tuned\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.917692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsz52\" (UniqueName: \"kubernetes.io/projected/7e23b525-fa64-4588-a489-25c0857716b4-kube-api-access-nsz52\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.917692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.917692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20690bdd-9445-4a57-a6a7-87a7c0cb824f-host-slash\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:02.917692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-kubelet\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.917692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-etc-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07df0fd4-1644-4f1c-8de2-466562163d1b-tmp\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7-agent-certs\") pod \"konnectivity-agent-kwslh\" (UID: \"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7\") " pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-tmp-dir\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-cni-multus\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-cni-netd\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-hosts-file\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-registration-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30dc45cb-9ef0-411a-88b1-2673963e5977-cni-binary-copy\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.917973 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.917971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-k8s-cni-cncf-io\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-hostroot\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b1989d-a634-4622-82c7-9ba42c0b829b-host\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2p8f\" (UniqueName: \"kubernetes.io/projected/50933fa6-d2ac-4292-ba25-af08d8d200ba-kube-api-access-c2p8f\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-conf-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-dbus\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7-konnectivity-ca\") pod \"konnectivity-agent-kwslh\" (UID: \"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7\") " pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-cni-bin\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-system-cni-dir\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-device-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-multus-certs\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-cnibin\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.918298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-node-log\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-cnibin\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-system-cni-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-systemd\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-log-socket\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-kubelet-config\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-systemd\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-lib-modules\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20690bdd-9445-4a57-a6a7-87a7c0cb824f-iptables-alerter-script\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-cni-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-var-lib-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq5zf\" (UniqueName: \"kubernetes.io/projected/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-kube-api-access-kq5zf\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-kubernetes\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5xpf\" (UniqueName: \"kubernetes.io/projected/07df0fd4-1644-4f1c-8de2-466562163d1b-kube-api-access-d5xpf\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-sys-fs\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.918775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjkq\" (UniqueName: \"kubernetes.io/projected/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-kube-api-access-nqjkq\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlpq\" (UniqueName: \"kubernetes.io/projected/d8b1989d-a634-4622-82c7-9ba42c0b829b-kube-api-access-hwlpq\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovn-node-metrics-cert\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysctl-conf\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-var-lib-kubelet\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-modprobe-d\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-sys\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-host\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-cni-bin\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b1989d-a634-4622-82c7-9ba42c0b829b-serviceca\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-env-overrides\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82ww7\" (UniqueName: \"kubernetes.io/projected/20690bdd-9445-4a57-a6a7-87a7c0cb824f-kube-api-access-82ww7\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.918996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-netns\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-daemon-config\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.919377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-systemd-units\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysconfig\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-socket-dir-parent\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-slash\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-run\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-kubelet\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:02.919936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:02.919287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovnkube-config\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.016624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.016538    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:03.019730 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-cni-bin\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.019730 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b1989d-a634-4622-82c7-9ba42c0b829b-serviceca\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:03.019930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-env-overrides\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.019930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-cni-bin\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.019930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82ww7\" (UniqueName: \"kubernetes.io/projected/20690bdd-9445-4a57-a6a7-87a7c0cb824f-kube-api-access-82ww7\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:03.019930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-netns\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.020102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-daemon-config\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.020102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-systemd-units\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.019980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysconfig\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.020229 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-netns\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.020229 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-systemd-units\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020303 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysconfig\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.020303 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b1989d-a634-4622-82c7-9ba42c0b829b-serviceca\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:03.020303 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-env-overrides\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-socket-dir-parent\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-socket-dir-parent\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-slash\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-run\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-slash\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-kubelet\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-run\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-kubelet\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovnkube-config\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-daemon-config\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovnkube-script-lib\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysctl-d\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hc2\" (UniqueName: \"kubernetes.io/projected/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-kube-api-access-j9hc2\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-ovn\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysctl-d\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-ovn\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.020879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-socket-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-etc-kubernetes\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9bm\" (UniqueName: \"kubernetes.io/projected/30dc45cb-9ef0-411a-88b1-2673963e5977-kube-api-access-dq9bm\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-run-netns\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-tuned\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-socket-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-etc-kubernetes\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.020993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsz52\" (UniqueName: \"kubernetes.io/projected/7e23b525-fa64-4588-a489-25c0857716b4-kube-api-access-nsz52\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20690bdd-9445-4a57-a6a7-87a7c0cb824f-host-slash\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-kubelet\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-etc-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-run-netns\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovnkube-config\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20690bdd-9445-4a57-a6a7-87a7c0cb824f-host-slash\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07df0fd4-1644-4f1c-8de2-466562163d1b-tmp\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7-agent-certs\") pod \"konnectivity-agent-kwslh\" (UID: \"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7\") " pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:03.021679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovnkube-script-lib\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-kubelet\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-tmp-dir\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-etc-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-cni-multus\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-cni-netd\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-var-lib-cni-multus\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.021450    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-hosts-file\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-cni-netd\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-registration-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-hosts-file\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021512    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 00:43:03.022671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-registration-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.021583    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:03.52150289 +0000 UTC m=+2.064639853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30dc45cb-9ef0-411a-88b1-2673963e5977-cni-binary-copy\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-tmp-dir\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-k8s-cni-cncf-io\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-hostroot\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-k8s-cni-cncf-io\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b1989d-a634-4622-82c7-9ba42c0b829b-host\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-hostroot\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2p8f\" (UniqueName: \"kubernetes.io/projected/50933fa6-d2ac-4292-ba25-af08d8d200ba-kube-api-access-c2p8f\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b1989d-a634-4622-82c7-9ba42c0b829b-host\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-conf-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-dbus\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-conf-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7-konnectivity-ca\") pod \"konnectivity-agent-kwslh\" (UID: \"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7\") " pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-dbus\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-cni-bin\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.023643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-system-cni-dir\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.021980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-system-cni-dir\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-device-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-device-dir\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-multus-certs\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30dc45cb-9ef0-411a-88b1-2673963e5977-cni-binary-copy\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-cnibin\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-node-log\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-host-run-multus-certs\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-cnibin\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-cni-bin\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-system-cni-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-node-log\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-systemd\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-log-socket\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.024403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-kubelet-config\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-systemd\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-lib-modules\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20690bdd-9445-4a57-a6a7-87a7c0cb824f-iptables-alerter-script\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-cni-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-var-lib-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq5zf\" (UniqueName: \"kubernetes.io/projected/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-kube-api-access-kq5zf\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-kubernetes\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xpf\" (UniqueName: \"kubernetes.io/projected/07df0fd4-1644-4f1c-8de2-466562163d1b-kube-api-access-d5xpf\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-sys-fs\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjkq\" (UniqueName: \"kubernetes.io/projected/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-kube-api-access-nqjkq\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-systemd\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlpq\" (UniqueName: \"kubernetes.io/projected/d8b1989d-a634-4622-82c7-9ba42c0b829b-kube-api-access-hwlpq\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovn-node-metrics-cert\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysctl-conf\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-var-lib-kubelet\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.025365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-modprobe-d\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-sys\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-lib-modules\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-host\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-host\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-sys-fs\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-multus-cni-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-var-lib-openvswitch\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.022653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7-konnectivity-ca\") pod \"konnectivity-agent-kwslh\" (UID: \"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7\") " pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-kubernetes\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50933fa6-d2ac-4292-ba25-af08d8d200ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50933fa6-d2ac-4292-ba25-af08d8d200ba-cnibin\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-run-systemd\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-log-socket\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-system-cni-dir\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.026185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30dc45cb-9ef0-411a-88b1-2673963e5977-cnibin\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e23b525-fa64-4588-a489-25c0857716b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-kubelet-config\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-var-lib-kubelet\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.023519    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.023592    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret podName:dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb nodeName:}" failed. No retries permitted until 2026-09-16 00:43:03.523573214 +0000 UTC m=+2.066710183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret") pod "global-pull-secret-syncer-pxhn7" (UID: "dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-sys\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-modprobe-d\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20690bdd-9445-4a57-a6a7-87a7c0cb824f-iptables-alerter-script\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.023692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-sysctl-conf\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.025148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07df0fd4-1644-4f1c-8de2-466562163d1b-etc-tuned\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.025179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07df0fd4-1644-4f1c-8de2-466562163d1b-tmp\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.025529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7-agent-certs\") pod \"konnectivity-agent-kwslh\" (UID: \"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7\") " pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:03.026804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.026273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-ovn-node-metrics-cert\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.027258 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.027001    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:03.027258 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.027019    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:03.027258 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.027030    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod openshift-network-diagnostics/network-check-target-h6x8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:03.027258 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.027116    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7 podName:eab2796e-f237-4343-a84e-c7393f625e1b nodeName:}" failed. No retries permitted until 2026-09-16 00:43:03.527100557 +0000 UTC m=+2.070237512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7") pod "network-check-target-h6x8j" (UID: "eab2796e-f237-4343-a84e-c7393f625e1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:03.028126 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.028105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82ww7\" (UniqueName: \"kubernetes.io/projected/20690bdd-9445-4a57-a6a7-87a7c0cb824f-kube-api-access-82ww7\") pod \"iptables-alerter-vwb6j\" (UID: \"20690bdd-9445-4a57-a6a7-87a7c0cb824f\") " pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:03.029676 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.029652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hc2\" (UniqueName: \"kubernetes.io/projected/6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405-kube-api-access-j9hc2\") pod \"node-resolver-46zrq\" (UID: \"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405\") " pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:03.030278 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.030248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9bm\" (UniqueName: \"kubernetes.io/projected/30dc45cb-9ef0-411a-88b1-2673963e5977-kube-api-access-dq9bm\") pod \"multus-v9fk6\" (UID: \"30dc45cb-9ef0-411a-88b1-2673963e5977\") " pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.032807 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.032605    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsz52\" (UniqueName: \"kubernetes.io/projected/7e23b525-fa64-4588-a489-25c0857716b4-kube-api-access-nsz52\") pod \"aws-ebs-csi-driver-node-khm4z\" (UID: \"7e23b525-fa64-4588-a489-25c0857716b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.033237 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.033216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2p8f\" (UniqueName: \"kubernetes.io/projected/50933fa6-d2ac-4292-ba25-af08d8d200ba-kube-api-access-c2p8f\") pod \"multus-additional-cni-plugins-x62zx\" (UID: \"50933fa6-d2ac-4292-ba25-af08d8d200ba\") " pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.033391 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.033371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlpq\" (UniqueName: \"kubernetes.io/projected/d8b1989d-a634-4622-82c7-9ba42c0b829b-kube-api-access-hwlpq\") pod \"node-ca-8z8zz\" (UID: \"d8b1989d-a634-4622-82c7-9ba42c0b829b\") " pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:03.033969 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.033947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq5zf\" (UniqueName: \"kubernetes.io/projected/c2a5740b-6a58-4baa-b4af-6b8caaf9c71d-kube-api-access-kq5zf\") pod \"ovnkube-node-m5btj\" (UID: \"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.034368 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.034350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjkq\" (UniqueName: \"kubernetes.io/projected/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-kube-api-access-nqjkq\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:03.034368 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.034361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xpf\" (UniqueName: \"kubernetes.io/projected/07df0fd4-1644-4f1c-8de2-466562163d1b-kube-api-access-d5xpf\") pod \"tuned-fm86s\" (UID: \"07df0fd4-1644-4f1c-8de2-466562163d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.134827 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.134785    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3855482ecf545e291be557b4f332fc2e.slice/crio-149d80ca2b4a107374659273308d56d9dec8e8b0eb5f7c4c9dd5438a43aee7c7 WatchSource:0}: Error finding container 149d80ca2b4a107374659273308d56d9dec8e8b0eb5f7c4c9dd5438a43aee7c7: Status 404 returned error can't find the container with id 149d80ca2b4a107374659273308d56d9dec8e8b0eb5f7c4c9dd5438a43aee7c7
Sep 16 00:43:03.139037 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.139017    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:43:03.142873 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.142853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:03.151423 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.151399    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48fcf55_35a0_4c4b_a7c8_2a5ecaefc7e7.slice/crio-f44915abcac75cfa477519765ebf9735152f32846cdb00364b0975332a79e170 WatchSource:0}: Error finding container f44915abcac75cfa477519765ebf9735152f32846cdb00364b0975332a79e170: Status 404 returned error can't find the container with id f44915abcac75cfa477519765ebf9735152f32846cdb00364b0975332a79e170
Sep 16 00:43:03.161506 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.161470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-46zrq"
Sep 16 00:43:03.169577 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.169538    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebf9614_24d3_4cc7_8fb9_a1d6c3d92405.slice/crio-335230c878c79f252eee38d0bb1c9939784a3a42e915531f98b64aa455130ccd WatchSource:0}: Error finding container 335230c878c79f252eee38d0bb1c9939784a3a42e915531f98b64aa455130ccd: Status 404 returned error can't find the container with id 335230c878c79f252eee38d0bb1c9939784a3a42e915531f98b64aa455130ccd
Sep 16 00:43:03.181994 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.181973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8z8zz"
Sep 16 00:43:03.183084 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.183067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x62zx"
Sep 16 00:43:03.192819 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.192794    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b1989d_a634_4622_82c7_9ba42c0b829b.slice/crio-4432d899725aebc50cc20a182f06542d8e4aea476e73d19a43f186b5cfdfdc16 WatchSource:0}: Error finding container 4432d899725aebc50cc20a182f06542d8e4aea476e73d19a43f186b5cfdfdc16: Status 404 returned error can't find the container with id 4432d899725aebc50cc20a182f06542d8e4aea476e73d19a43f186b5cfdfdc16
Sep 16 00:43:03.193259 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.193236    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50933fa6_d2ac_4292_ba25_af08d8d200ba.slice/crio-44b328d17dfcb17dd66f999fd5361a2d4081823336109929aa117b8e4c5fa69c WatchSource:0}: Error finding container 44b328d17dfcb17dd66f999fd5361a2d4081823336109929aa117b8e4c5fa69c: Status 404 returned error can't find the container with id 44b328d17dfcb17dd66f999fd5361a2d4081823336109929aa117b8e4c5fa69c
Sep 16 00:43:03.214587 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.214554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vwb6j"
Sep 16 00:43:03.224163 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.224138    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20690bdd_9445_4a57_a6a7_87a7c0cb824f.slice/crio-0d945ad57061a45e777d3c0124fc25d05a000d6e5fc066e3420a88bb2f36c8f0 WatchSource:0}: Error finding container 0d945ad57061a45e777d3c0124fc25d05a000d6e5fc066e3420a88bb2f36c8f0: Status 404 returned error can't find the container with id 0d945ad57061a45e777d3c0124fc25d05a000d6e5fc066e3420a88bb2f36c8f0
Sep 16 00:43:03.244588 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.244559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z"
Sep 16 00:43:03.252454 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.252426    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e23b525_fa64_4588_a489_25c0857716b4.slice/crio-6dac9aa46c8ba506c350b1c6eb842f3598abb4f713926475dab7c909d0438a19 WatchSource:0}: Error finding container 6dac9aa46c8ba506c350b1c6eb842f3598abb4f713926475dab7c909d0438a19: Status 404 returned error can't find the container with id 6dac9aa46c8ba506c350b1c6eb842f3598abb4f713926475dab7c909d0438a19
Sep 16 00:43:03.258712 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.258693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9fk6"
Sep 16 00:43:03.266427 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.266406    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30dc45cb_9ef0_411a_88b1_2673963e5977.slice/crio-5a2c6178495f22232e8bd100d944846c82967c0293242f948501ed4869c060c2 WatchSource:0}: Error finding container 5a2c6178495f22232e8bd100d944846c82967c0293242f948501ed4869c060c2: Status 404 returned error can't find the container with id 5a2c6178495f22232e8bd100d944846c82967c0293242f948501ed4869c060c2
Sep 16 00:43:03.274048 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.274027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:03.279182 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.279166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fm86s"
Sep 16 00:43:03.281593 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.281565    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a5740b_6a58_4baa_b4af_6b8caaf9c71d.slice/crio-1542e80a6ecf51e77b863ffd3327a97887e5f6c9c2b317f75ac9c8335959eb13 WatchSource:0}: Error finding container 1542e80a6ecf51e77b863ffd3327a97887e5f6c9c2b317f75ac9c8335959eb13: Status 404 returned error can't find the container with id 1542e80a6ecf51e77b863ffd3327a97887e5f6c9c2b317f75ac9c8335959eb13
Sep 16 00:43:03.286578 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.286530    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07df0fd4_1644_4f1c_8de2_466562163d1b.slice/crio-61170220c18fbd49bf2870a1be97881822a29f134e00a730bbc4f5628d177e61 WatchSource:0}: Error finding container 61170220c18fbd49bf2870a1be97881822a29f134e00a730bbc4f5628d177e61: Status 404 returned error can't find the container with id 61170220c18fbd49bf2870a1be97881822a29f134e00a730bbc4f5628d177e61
Sep 16 00:43:03.323719 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:03.323689    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b215942334fadb1e5657317484eaae0.slice/crio-6a075edbf3fb6c9fdf1b49129503c0492ba366154a09566de5616f27d8fbd0e6 WatchSource:0}: Error finding container 6a075edbf3fb6c9fdf1b49129503c0492ba366154a09566de5616f27d8fbd0e6: Status 404 returned error can't find the container with id 6a075edbf3fb6c9fdf1b49129503c0492ba366154a09566de5616f27d8fbd0e6
Sep 16 00:43:03.325514 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.325492    2581 kuberuntime_manager.go:1459] "Unhandled Error" err="init container setup start failed in pod kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal_openshift-machine-config-operator(2b215942334fadb1e5657317484eaae0): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 16 00:43:03.325628 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.325527    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal" podUID="2b215942334fadb1e5657317484eaae0"
Sep 16 00:43:03.526350 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.526315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:03.526580 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.526372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:03.526580 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.526463    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:03.526580 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.526464    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:03.526580 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.526539    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret podName:dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb nodeName:}" failed. No retries permitted until 2026-09-16 00:43:04.526524805 +0000 UTC m=+3.069661773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret") pod "global-pull-secret-syncer-pxhn7" (UID: "dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:03.526580 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.526555    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:04.526548541 +0000 UTC m=+3.069685494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:03.627350 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.627309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:03.627551 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.627472    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:03.627551 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.627521    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:03.627551 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.627537    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod openshift-network-diagnostics/network-check-target-h6x8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:03.627739 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:03.627626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7 podName:eab2796e-f237-4343-a84e-c7393f625e1b nodeName:}" failed. No retries permitted until 2026-09-16 00:43:04.627588403 +0000 UTC m=+3.170725374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7") pod "network-check-target-h6x8j" (UID: "eab2796e-f237-4343-a84e-c7393f625e1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:03.865796 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.865618    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:38:03 +0000 UTC" deadline="2028-03-18 13:34:38.693077004 +0000 UTC"
Sep 16 00:43:03.865796 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.865683    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13188h51m34.827398888s"
Sep 16 00:43:03.973403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.973277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerStarted","Data":"44b328d17dfcb17dd66f999fd5361a2d4081823336109929aa117b8e4c5fa69c"}
Sep 16 00:43:03.982776 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.982683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8z8zz" event={"ID":"d8b1989d-a634-4622-82c7-9ba42c0b829b","Type":"ContainerStarted","Data":"4432d899725aebc50cc20a182f06542d8e4aea476e73d19a43f186b5cfdfdc16"}
Sep 16 00:43:03.992043 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:03.992007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-46zrq" event={"ID":"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405","Type":"ContainerStarted","Data":"335230c878c79f252eee38d0bb1c9939784a3a42e915531f98b64aa455130ccd"}
Sep 16 00:43:04.004975 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.004936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fm86s" event={"ID":"07df0fd4-1644-4f1c-8de2-466562163d1b","Type":"ContainerStarted","Data":"61170220c18fbd49bf2870a1be97881822a29f134e00a730bbc4f5628d177e61"}
Sep 16 00:43:04.016699 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.016658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"1542e80a6ecf51e77b863ffd3327a97887e5f6c9c2b317f75ac9c8335959eb13"}
Sep 16 00:43:04.016991 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.016970    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:04.024215 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.022468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vwb6j" event={"ID":"20690bdd-9445-4a57-a6a7-87a7c0cb824f","Type":"ContainerStarted","Data":"0d945ad57061a45e777d3c0124fc25d05a000d6e5fc066e3420a88bb2f36c8f0"}
Sep 16 00:43:04.027957 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.027923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwslh" event={"ID":"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7","Type":"ContainerStarted","Data":"f44915abcac75cfa477519765ebf9735152f32846cdb00364b0975332a79e170"}
Sep 16 00:43:04.044928 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.044882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal" event={"ID":"3855482ecf545e291be557b4f332fc2e","Type":"ContainerStarted","Data":"149d80ca2b4a107374659273308d56d9dec8e8b0eb5f7c4c9dd5438a43aee7c7"}
Sep 16 00:43:04.063914 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.063246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal" event={"ID":"2b215942334fadb1e5657317484eaae0","Type":"ContainerStarted","Data":"6a075edbf3fb6c9fdf1b49129503c0492ba366154a09566de5616f27d8fbd0e6"}
Sep 16 00:43:04.068717 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.068046    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal" podUID="2b215942334fadb1e5657317484eaae0"
Sep 16 00:43:04.071284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.071195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9fk6" event={"ID":"30dc45cb-9ef0-411a-88b1-2673963e5977","Type":"ContainerStarted","Data":"5a2c6178495f22232e8bd100d944846c82967c0293242f948501ed4869c060c2"}
Sep 16 00:43:04.109211 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.109170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z" event={"ID":"7e23b525-fa64-4588-a489-25c0857716b4","Type":"ContainerStarted","Data":"6dac9aa46c8ba506c350b1c6eb842f3598abb4f713926475dab7c909d0438a19"}
Sep 16 00:43:04.509692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.509661    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:04.533153 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.533113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:04.533371 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.533195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:04.533371 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.533359    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:04.533494 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.533477    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:06.533408417 +0000 UTC m=+5.076545395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:04.533949 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.533923    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:04.534066 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.533982    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret podName:dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb nodeName:}" failed. No retries permitted until 2026-09-16 00:43:06.533965502 +0000 UTC m=+5.077102477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret") pod "global-pull-secret-syncer-pxhn7" (UID: "dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:04.635793 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.633635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:04.635793 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.633803    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:04.635793 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.633827    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:04.635793 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.633840    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod openshift-network-diagnostics/network-check-target-h6x8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:04.635793 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.633901    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7 podName:eab2796e-f237-4343-a84e-c7393f625e1b nodeName:}" failed. No retries permitted until 2026-09-16 00:43:06.633881849 +0000 UTC m=+5.177018810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7") pod "network-check-target-h6x8j" (UID: "eab2796e-f237-4343-a84e-c7393f625e1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:04.866723 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.866616    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:38:03 +0000 UTC" deadline="2028-04-26 00:24:49.529319069 +0000 UTC"
Sep 16 00:43:04.866723 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.866680    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14111h41m44.662643409s"
Sep 16 00:43:04.935826 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.935792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:04.936013 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.935915    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:04.936363 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.936336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:04.936531 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.936476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:04.936628 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:04.936549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:04.936692 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:04.936675    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:05.119787 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:05.119746    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal" podUID="2b215942334fadb1e5657317484eaae0"
Sep 16 00:43:06.550870 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:06.550609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:06.550870 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:06.550690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:06.550870 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.550821    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:06.551471 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.550891    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret podName:dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb nodeName:}" failed. No retries permitted until 2026-09-16 00:43:10.550872701 +0000 UTC m=+9.094009656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret") pod "global-pull-secret-syncer-pxhn7" (UID: "dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:06.551471 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.550821    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:06.551471 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.551291    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:10.551271188 +0000 UTC m=+9.094408160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:06.651429 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:06.651006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:06.651429 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.651329    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:06.651429 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.651358    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:06.651429 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.651373    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod openshift-network-diagnostics/network-check-target-h6x8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:06.651429 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.651432    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7 podName:eab2796e-f237-4343-a84e-c7393f625e1b nodeName:}" failed. No retries permitted until 2026-09-16 00:43:10.651412737 +0000 UTC m=+9.194549718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7") pod "network-check-target-h6x8j" (UID: "eab2796e-f237-4343-a84e-c7393f625e1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:06.935705 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:06.934841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:06.935705 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.935012    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:06.935705 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:06.935435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:06.935705 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.935517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:06.935705 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:06.935573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:06.935705 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:06.935618    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:08.935242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:08.934767    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:08.935242 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:08.934903    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:08.935242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:08.935017    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:08.935242 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:08.935152    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:08.935242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:08.935174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:08.935872 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:08.935313    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:10.586156 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:10.585858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:10.586156 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:10.585954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:10.586156 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.586014    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:10.586156 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.586068    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:10.586156 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.586099    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret podName:dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb nodeName:}" failed. No retries permitted until 2026-09-16 00:43:18.58607813 +0000 UTC m=+17.129215091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret") pod "global-pull-secret-syncer-pxhn7" (UID: "dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:10.586156 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.586127    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:18.58610744 +0000 UTC m=+17.129244397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:10.686802 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:10.686760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:10.687047 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.686997    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:10.687047 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.687024    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:10.687047 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.687038    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod openshift-network-diagnostics/network-check-target-h6x8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:10.687316 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.687100    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7 podName:eab2796e-f237-4343-a84e-c7393f625e1b nodeName:}" failed. No retries permitted until 2026-09-16 00:43:18.687080986 +0000 UTC m=+17.230217946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7") pod "network-check-target-h6x8j" (UID: "eab2796e-f237-4343-a84e-c7393f625e1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:10.934793 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:10.934713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:10.934793 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:10.934753    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:10.935008 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:10.934714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:10.935008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.934858    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:10.935008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.934965    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:10.935156 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:10.935052    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:12.935524 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:12.935469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:12.936009 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:12.935472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:12.936009 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:12.935613    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:12.936009 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:12.935730    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:12.936009 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:12.935472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:12.936009 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:12.935847    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:14.934908 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:14.934871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:14.935385 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:14.934885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:14.935385 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:14.934999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:14.935385 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:14.935099    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:14.935385 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:14.934871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:14.935385 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:14.935186    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:16.935221 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:16.935179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:16.935221 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:16.935204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:16.935221 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:16.935227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:16.935789 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:16.935319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:16.935789 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:16.935409    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:16.935789 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:16.935730    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:18.645700 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:18.645535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:18.645700 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:18.645610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:18.646221 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.645763    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:18.646221 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.645827    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret podName:dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb nodeName:}" failed. No retries permitted until 2026-09-16 00:43:34.645807827 +0000 UTC m=+33.188944794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret") pod "global-pull-secret-syncer-pxhn7" (UID: "dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:18.646221 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.645900    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:18.646221 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.645935    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:34.64592338 +0000 UTC m=+33.189060346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:18.746761 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:18.746677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:18.746918 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.746875    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:18.746918 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.746904    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:18.746918 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.746919    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod openshift-network-diagnostics/network-check-target-h6x8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:18.747054 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.746986    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7 podName:eab2796e-f237-4343-a84e-c7393f625e1b nodeName:}" failed. No retries permitted until 2026-09-16 00:43:34.746968153 +0000 UTC m=+33.290105123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7") pod "network-check-target-h6x8j" (UID: "eab2796e-f237-4343-a84e-c7393f625e1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:18.935516 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:18.935465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:18.935516 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:18.935517    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:18.935858 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:18.935465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:18.935858 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.935631    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:18.935858 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.935777    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:18.935858 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:18.935852    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:20.934702 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:20.934663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:20.934702 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:20.934685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:20.935368 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:20.934713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:20.935368 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:20.934799    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:20.935368 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:20.934868    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:20.935368 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:20.934933    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:22.937535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:22.935687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:22.937535 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:22.936707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:22.937535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:22.936793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:22.937535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:22.936859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:22.937535 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:22.936958    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:22.937535 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:22.937040    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:23.151592 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.151344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z" event={"ID":"7e23b525-fa64-4588-a489-25c0857716b4","Type":"ContainerStarted","Data":"2ce80c8e9f6e939a096b89127301a329bc8ceaaa78be8a893ee784deb5ebef0d"}
Sep 16 00:43:23.153118 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.153089    2581 generic.go:356] "Generic (PLEG): container finished" podID="50933fa6-d2ac-4292-ba25-af08d8d200ba" containerID="a92a98449231e93809facd2d7875f6725f19ad402e07a710e7cb6806146a7b46" exitCode=0
Sep 16 00:43:23.153219 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.153182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerDied","Data":"a92a98449231e93809facd2d7875f6725f19ad402e07a710e7cb6806146a7b46"}
Sep 16 00:43:23.154839 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.154819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8z8zz" event={"ID":"d8b1989d-a634-4622-82c7-9ba42c0b829b","Type":"ContainerStarted","Data":"36c1bc368f52143e43b5655827cb4c712c28dc50ef2ac59ecc22279c5a25820d"}
Sep 16 00:43:23.156547 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.156513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-46zrq" event={"ID":"6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405","Type":"ContainerStarted","Data":"816830e847c393fd0c640ea2e4c895b0d3ea3659336dc06976b83da784572432"}
Sep 16 00:43:23.158008 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.157981    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fm86s" event={"ID":"07df0fd4-1644-4f1c-8de2-466562163d1b","Type":"ContainerStarted","Data":"59c8103d2ef9061b90f65a2424595c399f2a99381695a7b733e12075ebaf588d"}
Sep 16 00:43:23.160071 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.160047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"b986e8aab75d3100501e2ea94cfe8aee4d7b08532209ba1bb848ac8472e459f4"}
Sep 16 00:43:23.160153 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.160113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"780ba424b1d2fa64f3577135eacb9c8528460cc50b6eacae313db4dcccddfe41"}
Sep 16 00:43:23.161443 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.161426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwslh" event={"ID":"d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7","Type":"ContainerStarted","Data":"3a74987c25bfb5be4dd711e49b09f66276b0035ab6cbe6ce3bb441ecd440126c"}
Sep 16 00:43:23.163691 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.163672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal" event={"ID":"3855482ecf545e291be557b4f332fc2e","Type":"ContainerStarted","Data":"fb7190af4b10ae0926e5054077683e5f2732fd0679221923fed5a862aac6b604"}
Sep 16 00:43:23.165174 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.165154    2581 generic.go:356] "Generic (PLEG): container finished" podID="2b215942334fadb1e5657317484eaae0" containerID="7a28d525273ded03bd02f7103a77281d15fe08d9486028d2d1ca4e491040befb" exitCode=0
Sep 16 00:43:23.165260 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.165224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal" event={"ID":"2b215942334fadb1e5657317484eaae0","Type":"ContainerDied","Data":"7a28d525273ded03bd02f7103a77281d15fe08d9486028d2d1ca4e491040befb"}
Sep 16 00:43:23.169845 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.169818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9fk6" event={"ID":"30dc45cb-9ef0-411a-88b1-2673963e5977","Type":"ContainerStarted","Data":"d92e893362aee739ee9338e5dc9666bdb9e51f2df8901cac51ac999314d4d653"}
Sep 16 00:43:23.186887 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.186840    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fm86s" podStartSLOduration=1.146214014 podStartE2EDuration="20.186824184s" totalImagesPullingTime="19.04061017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:23.186277065 +0000 UTC m=+21.729414040" watchObservedRunningTime="2026-09-16 00:43:23.186824184 +0000 UTC m=+21.729961159"
Sep 16 00:43:23.221136 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.220999    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8z8zz" podStartSLOduration=2.087267398 podStartE2EDuration="21.220975719s" totalImagesPullingTime="19.133708321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:23.198278157 +0000 UTC m=+21.741415133" watchObservedRunningTime="2026-09-16 00:43:23.220975719 +0000 UTC m=+21.764112695"
Sep 16 00:43:23.257133 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.257088    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v9fk6" podStartSLOduration=1.692207255 podStartE2EDuration="21.257071076s" totalImagesPullingTime="19.564863821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:23.221374611 +0000 UTC m=+21.764511587" watchObservedRunningTime="2026-09-16 00:43:23.257071076 +0000 UTC m=+21.800208050"
Sep 16 00:43:23.280036 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.279980    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-4.ec2.internal" podStartSLOduration=20.279957417 podStartE2EDuration="20.279957417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:23.278815054 +0000 UTC m=+21.821952029" watchObservedRunningTime="2026-09-16 00:43:23.279957417 +0000 UTC m=+21.823094393"
Sep 16 00:43:23.293133 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.293081    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kwslh" podStartSLOduration=2.117513687 podStartE2EDuration="21.293061839s" totalImagesPullingTime="19.175548152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:23.293043444 +0000 UTC m=+21.836180421" watchObservedRunningTime="2026-09-16 00:43:23.293061839 +0000 UTC m=+21.836198815"
Sep 16 00:43:23.309946 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:23.309896    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-46zrq" podStartSLOduration=2.137679737 podStartE2EDuration="21.309874066s" totalImagesPullingTime="19.172194329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:23.309457286 +0000 UTC m=+21.852594262" watchObservedRunningTime="2026-09-16 00:43:23.309874066 +0000 UTC m=+21.853011040"
Sep 16 00:43:24.175428 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.175385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"3f0cbd660551da8540345313f60fc8193016094ff050cbc2a801f234cfd0867d"}
Sep 16 00:43:24.176118 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.175494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"a888a4a02543893425ce686b360e074ec04cdc5198186f67af9ad136000e2d4d"}
Sep 16 00:43:24.176118 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.175550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"70c5eb06d4c668d3c206baf79281cbb9691bacca50b1128c6919b5ec7c5e4a2d"}
Sep 16 00:43:24.176118 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.175597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"3521f505273ad154a3bcae8d0eb3cddf1af1f991895b1a312102faa00f2331c9"}
Sep 16 00:43:24.177613 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.177582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal" event={"ID":"2b215942334fadb1e5657317484eaae0","Type":"ContainerStarted","Data":"10c1795ca286599c2d181532540c25f52e63f7f2890b447fe515326304fd5d9b"}
Sep 16 00:43:24.192156 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.192105    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-4.ec2.internal" podStartSLOduration=21.192088044 podStartE2EDuration="21.192088044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:24.191629892 +0000 UTC m=+22.734766868" watchObservedRunningTime="2026-09-16 00:43:24.192088044 +0000 UTC m=+22.735225020"
Sep 16 00:43:24.696902 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.696875    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 00:43:24.872834 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.872689    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T00:43:24.696893674Z","UUID":"9d0fa835-181d-4f05-81a7-d0cb5dd70ea2","Handler":null,"Name":"","Endpoint":""}
Sep 16 00:43:24.874698 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.874670    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 00:43:24.874847 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.874712    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 00:43:24.934914 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.934884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:24.935077 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.934985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:24.935077 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:24.934999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:24.935077 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:24.935015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:24.935261 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:24.935112    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:24.935261 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:24.935186    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:25.181991 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:25.181894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vwb6j" event={"ID":"20690bdd-9445-4a57-a6a7-87a7c0cb824f","Type":"ContainerStarted","Data":"a9850ecc4631b41cb4a63ec18938c59182963316e686e087893a2a9db7a136e9"}
Sep 16 00:43:25.185330 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:25.184880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z" event={"ID":"7e23b525-fa64-4588-a489-25c0857716b4","Type":"ContainerStarted","Data":"fc4fea5c65c611e27a6eefd9e4f4ec323a557a9ef7d82d70c7e75fedb7bfd5c6"}
Sep 16 00:43:25.195719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:25.195671    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vwb6j" podStartSLOduration=4.081056983 podStartE2EDuration="23.195654782s" totalImagesPullingTime="19.114597799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:25.195268525 +0000 UTC m=+23.738405499" watchObservedRunningTime="2026-09-16 00:43:25.195654782 +0000 UTC m=+23.738791758"
Sep 16 00:43:26.189252 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:26.189213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z" event={"ID":"7e23b525-fa64-4588-a489-25c0857716b4","Type":"ContainerStarted","Data":"749a7d1622e3b6722f6375388e8a2caa61156f6f3ba501a1f83599d42763ac8e"}
Sep 16 00:43:26.192622 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:26.192591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"8e33b5e06e18f7991f28f89df8689b656f2ff0bab55da8d1f5431b673512a707"}
Sep 16 00:43:26.205397 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:26.205350    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-khm4z" podStartSLOduration=2.586788336 podStartE2EDuration="24.205334762s" totalImagesPullingTime="21.618546426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:26.20478141 +0000 UTC m=+24.747918384" watchObservedRunningTime="2026-09-16 00:43:26.205334762 +0000 UTC m=+24.748471737"
Sep 16 00:43:26.935513 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:26.935468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:26.935703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:26.935469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:26.935703 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:26.935602    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:26.935703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:26.935469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:26.935703 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:26.935675    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:26.935918 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:26.935772    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:28.144160 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:28.144121    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:28.144898 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:28.144880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:28.935417 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:28.935245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:28.935642 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:28.935245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:28.935642 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:28.935508    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:28.935642 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:28.935599    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:28.935642 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:28.935245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:28.935816 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:28.935692    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:29.201761 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:29.201684    2581 generic.go:356] "Generic (PLEG): container finished" podID="50933fa6-d2ac-4292-ba25-af08d8d200ba" containerID="0283fda2c421b9a6fcad3d4f8ca5fa187f7dae2898361f415dc63f8c398eca6a" exitCode=0
Sep 16 00:43:29.202236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:29.201770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerDied","Data":"0283fda2c421b9a6fcad3d4f8ca5fa187f7dae2898361f415dc63f8c398eca6a"}
Sep 16 00:43:29.205612 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:29.205579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" event={"ID":"c2a5740b-6a58-4baa-b4af-6b8caaf9c71d","Type":"ContainerStarted","Data":"9d28798546f7405f0c0da1677102adb5b268f458d748d96cdd6645bce3ecff32"}
Sep 16 00:43:29.205868 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:29.205847    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:29.221277 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:29.221250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:29.245153 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:29.245106    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj" podStartSLOduration=7.689298194 podStartE2EDuration="27.245090028s" totalImagesPullingTime="19.555791834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:29.244908679 +0000 UTC m=+27.788045655" watchObservedRunningTime="2026-09-16 00:43:29.245090028 +0000 UTC m=+27.788227004"
Sep 16 00:43:30.141975 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.141945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxhn7"]
Sep 16 00:43:30.142235 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.142051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:30.142235 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:30.142137    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:30.145198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.145169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mftnx"]
Sep 16 00:43:30.145377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.145291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:30.145455 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:30.145389    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:30.145842 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.145814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h6x8j"]
Sep 16 00:43:30.145941 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.145913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:30.146020 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:30.145998    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:30.207888 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.207856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:30.208319 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.207926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:30.222344 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:30.222319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:43:31.211795 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:31.211758    2581 generic.go:356] "Generic (PLEG): container finished" podID="50933fa6-d2ac-4292-ba25-af08d8d200ba" containerID="22aaef6d6f661872b6935efc2f4fd46b9696e7e190cd7a37c62cfb0079ffea1b" exitCode=0
Sep 16 00:43:31.212282 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:31.211846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerDied","Data":"22aaef6d6f661872b6935efc2f4fd46b9696e7e190cd7a37c62cfb0079ffea1b"}
Sep 16 00:43:31.941262 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:31.941232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:31.941439 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:31.941233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:31.941439 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:31.941323    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:31.941439 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:31.941233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:31.941601 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:31.941448    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:31.941601 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:31.941513    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:32.330218 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:32.330174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:32.330682 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:32.330339    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 00:43:32.330938 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:32.330921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kwslh"
Sep 16 00:43:33.218276 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:33.218184    2581 generic.go:356] "Generic (PLEG): container finished" podID="50933fa6-d2ac-4292-ba25-af08d8d200ba" containerID="0420e12f6953e296deb0277c863f7364b583168069cb32e3bdc52dfdd60e8538" exitCode=0
Sep 16 00:43:33.218276 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:33.218265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerDied","Data":"0420e12f6953e296deb0277c863f7364b583168069cb32e3bdc52dfdd60e8538"}
Sep 16 00:43:33.935429 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:33.935401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:33.935429 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:33.935417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:33.936169 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:33.935402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:33.936169 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:33.935545    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:33.936169 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:33.935601    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:33.936169 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:33.935689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:34.664048 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:34.663860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:34.664048 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:34.663929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:34.664048 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.663992    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:34.664048 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.664011    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:34.664341 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.664064    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:44:06.664043648 +0000 UTC m=+65.207180618 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:34.664341 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.664082    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret podName:dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb nodeName:}" failed. No retries permitted until 2026-09-16 00:44:06.664073653 +0000 UTC m=+65.207210620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret") pod "global-pull-secret-syncer-pxhn7" (UID: "dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:34.764752 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:34.764709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:34.764932 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.764910    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:34.764981 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.764938    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:34.764981 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.764952    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod openshift-network-diagnostics/network-check-target-h6x8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:34.765057 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:34.765016    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7 podName:eab2796e-f237-4343-a84e-c7393f625e1b nodeName:}" failed. No retries permitted until 2026-09-16 00:44:06.764998702 +0000 UTC m=+65.308135661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7") pod "network-check-target-h6x8j" (UID: "eab2796e-f237-4343-a84e-c7393f625e1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:35.938659 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:35.938626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:35.939083 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:35.938741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:35.939083 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:35.938757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:35.939083 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:35.938753    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6x8j" podUID="eab2796e-f237-4343-a84e-c7393f625e1b"
Sep 16 00:43:35.939083 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:35.938847    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:43:35.939083 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:35.938895    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxhn7" podUID="dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb"
Sep 16 00:43:36.675307 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.675232    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-4.ec2.internal" event="NodeReady"
Sep 16 00:43:36.675461 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.675381    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 00:43:36.707063 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.707021    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"]
Sep 16 00:43:36.709881 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.709844    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78dfc67b84-z8qvb"]
Sep 16 00:43:36.710014 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.709982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:36.712027 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.712002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.712300 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.712276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gnmcb\""
Sep 16 00:43:36.712576 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.712555    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 00:43:36.712670 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.712591    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 00:43:36.714363 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.714342    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 00:43:36.714673 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.714652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 00:43:36.714936 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.714913    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 00:43:36.715098 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.715077    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l2glv\""
Sep 16 00:43:36.720176 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.719701    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hs5c9"]
Sep 16 00:43:36.720351 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.720330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 00:43:36.721988 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.721967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.724040 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.723999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"]
Sep 16 00:43:36.724358 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.724214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n5l29\""
Sep 16 00:43:36.724358 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.724220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 00:43:36.724358 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.724259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 00:43:36.724743 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.724724    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cj78c"]
Sep 16 00:43:36.727062 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.727036    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78dfc67b84-z8qvb"]
Sep 16 00:43:36.727062 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.727056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:36.729130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.729109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tzjjl\""
Sep 16 00:43:36.729354 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.729337    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 00:43:36.729559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.729543    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 00:43:36.729751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.729731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 00:43:36.741528 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.740729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cj78c"]
Sep 16 00:43:36.745444 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.744853    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hs5c9"]
Sep 16 00:43:36.779942 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.779908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-tmp-dir\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.780109 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.779966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-certificates\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780109 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmcn\" (UniqueName: \"kubernetes.io/projected/45b30463-b1d8-4f57-8f94-80411105dd84-kube-api-access-2bmcn\") pod \"ingress-canary-cj78c\" (UID: \"45b30463-b1d8-4f57-8f94-80411105dd84\") " pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:36.780109 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780109 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-config-volume\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.780276 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd6a62e3-f986-4971-8642-42861233c1b6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:36.780276 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-image-registry-private-configuration\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780276 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9n2\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-kube-api-access-wr9n2\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/339f2dfa-f2e9-4037-9bbd-84a53206887a-ca-trust-extracted\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780415 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:36.780415 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-installation-pull-secrets\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-bound-sa-token\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45b30463-b1d8-4f57-8f94-80411105dd84-cert\") pod \"ingress-canary-cj78c\" (UID: \"45b30463-b1d8-4f57-8f94-80411105dd84\") " pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:36.780538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-trusted-ca\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.780661 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.780661 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.780580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qll\" (UniqueName: \"kubernetes.io/projected/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-kube-api-access-b7qll\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.881729 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.881688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/339f2dfa-f2e9-4037-9bbd-84a53206887a-ca-trust-extracted\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.881910 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.881739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:36.881910 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.881770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-installation-pull-secrets\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.881910 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.881796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-bound-sa-token\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.881910 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.881820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45b30463-b1d8-4f57-8f94-80411105dd84-cert\") pod \"ingress-canary-cj78c\" (UID: \"45b30463-b1d8-4f57-8f94-80411105dd84\") " pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:36.881910 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.881850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-trusted-ca\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.881910 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.881876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.882261 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:36.881973    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:43:36.882261 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:36.882058    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:37.382035631 +0000 UTC m=+35.925172597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:43:36.882261 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/339f2dfa-f2e9-4037-9bbd-84a53206887a-ca-trust-extracted\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.882261 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qll\" (UniqueName: \"kubernetes.io/projected/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-kube-api-access-b7qll\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.882261 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-tmp-dir\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-certificates\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmcn\" (UniqueName: \"kubernetes.io/projected/45b30463-b1d8-4f57-8f94-80411105dd84-kube-api-access-2bmcn\") pod \"ingress-canary-cj78c\" (UID: \"45b30463-b1d8-4f57-8f94-80411105dd84\") " pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:36.882327    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-config-volume\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:36.882389    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:37.382372889 +0000 UTC m=+35.925509863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd6a62e3-f986-4971-8642-42861233c1b6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:36.882510 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-image-registry-private-configuration\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.882869 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9n2\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-kube-api-access-wr9n2\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.882869 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-trusted-ca\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.883198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-certificates\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.883198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.882978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-config-volume\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.883198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.883030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-tmp-dir\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.883198 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:36.883051    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:43:36.883198 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:36.883065    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:43:36.883198 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:36.883119    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:43:37.383105082 +0000 UTC m=+35.926242040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:43:36.883198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.883145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd6a62e3-f986-4971-8642-42861233c1b6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:36.886727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.886703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-image-registry-private-configuration\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.886727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.886719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45b30463-b1d8-4f57-8f94-80411105dd84-cert\") pod \"ingress-canary-cj78c\" (UID: \"45b30463-b1d8-4f57-8f94-80411105dd84\") " pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:36.887084 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.887067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-installation-pull-secrets\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.890514 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.890491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-bound-sa-token\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.890667 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.890644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qll\" (UniqueName: \"kubernetes.io/projected/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-kube-api-access-b7qll\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:36.891458 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.891432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9n2\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-kube-api-access-wr9n2\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:36.891593 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:36.891563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmcn\" (UniqueName: \"kubernetes.io/projected/45b30463-b1d8-4f57-8f94-80411105dd84-kube-api-access-2bmcn\") pod \"ingress-canary-cj78c\" (UID: \"45b30463-b1d8-4f57-8f94-80411105dd84\") " pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:37.045789 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.045729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:37.050557 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.050530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cj78c"
Sep 16 00:43:37.124905 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.124866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:37.387928 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.387832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:37.387928 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.387901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.387955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:37.387994    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:37.388059    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:37.388078    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:38.388055805 +0000 UTC m=+36.931192779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:37.388089    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:37.388104    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:37.388126    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:38.388109715 +0000 UTC m=+36.931246672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:43:37.388191 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:37.388152    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:43:38.388139458 +0000 UTC m=+36.931276410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:43:37.431122 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.431047    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:37.486932 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.486895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:37.934927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.934773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:43:37.935048 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.934773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:43:37.935291 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.934815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:43:37.938183 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.938158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 00:43:37.938302 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.938188    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 00:43:37.938302 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.938248    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6btht\""
Sep 16 00:43:37.938302 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.938252    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 00:43:37.938302 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.938172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 00:43:37.938468 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:37.938429    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ssxtd\""
Sep 16 00:43:38.047459 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:38.047372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cj78c"]
Sep 16 00:43:38.060394 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:43:38.060366    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b30463_b1d8_4f57_8f94_80411105dd84.slice/crio-f7c9417ea04af5a7df9e64c703b1a6a19628da2276a0b505848d515978855104 WatchSource:0}: Error finding container f7c9417ea04af5a7df9e64c703b1a6a19628da2276a0b505848d515978855104: Status 404 returned error can't find the container with id f7c9417ea04af5a7df9e64c703b1a6a19628da2276a0b505848d515978855104
Sep 16 00:43:38.236082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:38.236051    2581 generic.go:356] "Generic (PLEG): container finished" podID="50933fa6-d2ac-4292-ba25-af08d8d200ba" containerID="6c1cc382550956ffa289851ad5e4a2693397e90713170993eb5ac613ae881623" exitCode=0
Sep 16 00:43:38.236278 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:38.236127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerDied","Data":"6c1cc382550956ffa289851ad5e4a2693397e90713170993eb5ac613ae881623"}
Sep 16 00:43:38.237354 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:38.237330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cj78c" event={"ID":"45b30463-b1d8-4f57-8f94-80411105dd84","Type":"ContainerStarted","Data":"f7c9417ea04af5a7df9e64c703b1a6a19628da2276a0b505848d515978855104"}
Sep 16 00:43:38.395737 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:38.395706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:38.395898 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:38.395752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:38.395898 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:38.395808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:38.395898 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:38.395877    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:43:38.396018 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:38.395912    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:43:38.396018 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:38.395925    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:43:38.396018 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:38.395940    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:43:38.396018 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:38.395947    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:40.395928199 +0000 UTC m=+38.939065164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:43:38.396153 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:38.396021    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:43:40.396000728 +0000 UTC m=+38.939137688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:43:38.396153 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:38.396033    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:40.396026954 +0000 UTC m=+38.939163907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:43:39.243368 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:39.243327    2581 generic.go:356] "Generic (PLEG): container finished" podID="50933fa6-d2ac-4292-ba25-af08d8d200ba" containerID="f57740073af07419e8f0fb84ca17f54963bbb229d1054210439fbf23240000e6" exitCode=0
Sep 16 00:43:39.243990 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:39.243370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerDied","Data":"f57740073af07419e8f0fb84ca17f54963bbb229d1054210439fbf23240000e6"}
Sep 16 00:43:40.016414 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.016384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:40.116722 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.116689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:40.251367 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.251327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x62zx" event={"ID":"50933fa6-d2ac-4292-ba25-af08d8d200ba","Type":"ContainerStarted","Data":"108be9e49ac8b8388e6645c1cc78901a95ac726a11e9c10491b36e2569fcd96d"}
Sep 16 00:43:40.274021 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.273916    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x62zx" podStartSLOduration=6.659311336 podStartE2EDuration="38.273893155s" totalImagesPullingTime="31.614581819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:40.273379665 +0000 UTC m=+38.816516651" watchObservedRunningTime="2026-09-16 00:43:40.273893155 +0000 UTC m=+38.817030132"
Sep 16 00:43:40.392229 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.392193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:40.411215 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.411171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:40.411417 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:40.411298    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:43:40.411417 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:40.411378    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:44.411357525 +0000 UTC m=+42.954494501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:43:40.411575 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.411435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:40.411575 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.411509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:40.411676 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:40.411623    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:43:40.411676 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:40.411660    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:44.411647443 +0000 UTC m=+42.954784401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:43:40.411774 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:40.411727    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:43:40.411774 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:40.411738    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:43:40.411774 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:40.411772    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:43:44.411760741 +0000 UTC m=+42.954897697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:43:40.492847 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.492815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:40.609645 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.609541    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:40.662672 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:40.662629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:41.255369 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:41.255276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cj78c" event={"ID":"45b30463-b1d8-4f57-8f94-80411105dd84","Type":"ContainerStarted","Data":"2d2f93b96ab284f4885149c144adc5311ff85c39f49848a8662a0c11d9a558d0"}
Sep 16 00:43:41.271242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:41.271192    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cj78c" podStartSLOduration=2.436363795 podStartE2EDuration="5.271174046s" totalImagesPullingTime="2.834810251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:41.270051334 +0000 UTC m=+39.813188310" watchObservedRunningTime="2026-09-16 00:43:41.271174046 +0000 UTC m=+39.814311021"
Sep 16 00:43:44.279632 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:44.279601    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:44.335046 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:44.335010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:44.444206 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:44.444172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:44.444283 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:44.444225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:44.444283 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:44.444272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:44.444477 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:44.444320    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:43:44.444477 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:44.444357    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:43:44.444477 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:44.444391    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:52.444376367 +0000 UTC m=+50.987513343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:43:44.444477 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:44.444397    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:43:44.444477 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:44.444415    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:43:44.444477 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:44.444406    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:52.444400064 +0000 UTC m=+50.987537017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:43:44.444724 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:44.444511    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:43:52.44447092 +0000 UTC m=+50.987607880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:43:44.917352 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:44.917319    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:44.971217 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:44.971179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:52.507530 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:52.507468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:52.507585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:52.507619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:52.507636    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:52.507660    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:52.507726    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:44:08.507709461 +0000 UTC m=+67.050846452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:52.507727    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:52.507727    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:52.507784    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:44:08.507771145 +0000 UTC m=+67.050908102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:43:52.508008 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:43:52.507853    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:44:08.507833595 +0000 UTC m=+67.050970562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:43:57.204110 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:57.204076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:57.261667 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:57.261633    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:58.180699 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:58.180670    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:43:58.231951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:43:58.231916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:02.148166 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:02.148128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:02.223958 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:02.223926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m5btj"
Sep 16 00:44:02.374930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:02.374890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:02.427436 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:02.427369    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:06.156159 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.156125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:06.208130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.208086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:06.705540 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.705471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:44:06.705747 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.705589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:44:06.708087 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.708064    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 00:44:06.708087 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.708081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 00:44:06.715920 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:06.715896    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 00:44:06.716022 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:06.715957    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:45:10.715940658 +0000 UTC m=+129.259077625 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : secret "metrics-daemon-secret" not found
Sep 16 00:44:06.722915 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.722891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb-original-pull-secret\") pod \"global-pull-secret-syncer-pxhn7\" (UID: \"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb\") " pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:44:06.752568 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.752543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxhn7"
Sep 16 00:44:06.806074 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.806041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:44:06.815443 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.815414    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 00:44:06.818189 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.818166    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 00:44:06.829215 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:06.829192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/eab2796e-f237-4343-a84e-c7393f625e1b-kube-api-access-jtwk7\") pod \"network-check-target-h6x8j\" (UID: \"eab2796e-f237-4343-a84e-c7393f625e1b\") " pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:44:07.030818 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:07.030787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxhn7"]
Sep 16 00:44:07.037208 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:44:07.037174    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad4e663_1c7e_4bfc_8dc4_f7ec0c5d6ceb.slice/crio-8bdd173de0844e58bb5e9864e28d3cb0c69f281dfb5d36e0917157b47a93a6b0 WatchSource:0}: Error finding container 8bdd173de0844e58bb5e9864e28d3cb0c69f281dfb5d36e0917157b47a93a6b0: Status 404 returned error can't find the container with id 8bdd173de0844e58bb5e9864e28d3cb0c69f281dfb5d36e0917157b47a93a6b0
Sep 16 00:44:07.059344 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:07.059316    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ssxtd\""
Sep 16 00:44:07.068050 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:07.068030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:44:07.319311 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:07.319227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pxhn7" event={"ID":"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb","Type":"ContainerStarted","Data":"8bdd173de0844e58bb5e9864e28d3cb0c69f281dfb5d36e0917157b47a93a6b0"}
Sep 16 00:44:07.336768 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:07.336737    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h6x8j"]
Sep 16 00:44:07.340118 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:44:07.340093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab2796e_f237_4343_a84e_c7393f625e1b.slice/crio-1ffca390f6a1df16a3c9c1e23b741dd402d8cd1c3c9c34e05d9e2633cdad18bb WatchSource:0}: Error finding container 1ffca390f6a1df16a3c9c1e23b741dd402d8cd1c3c9c34e05d9e2633cdad18bb: Status 404 returned error can't find the container with id 1ffca390f6a1df16a3c9c1e23b741dd402d8cd1c3c9c34e05d9e2633cdad18bb
Sep 16 00:44:08.323874 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:08.323834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h6x8j" event={"ID":"eab2796e-f237-4343-a84e-c7393f625e1b","Type":"ContainerStarted","Data":"1ffca390f6a1df16a3c9c1e23b741dd402d8cd1c3c9c34e05d9e2633cdad18bb"}
Sep 16 00:44:08.517586 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:08.517543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:44:08.517783 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:08.517648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:44:08.517783 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:08.517693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:44:08.517783 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:08.517729    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:44:08.517783 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:08.517754    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:44:08.517985 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:08.517813    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:08.517985 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:08.517843    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:44:40.517821518 +0000 UTC m=+99.060958493 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:44:08.517985 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:08.517871    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:44:40.517855821 +0000 UTC m=+99.060992775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:08.517985 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:08.517813    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:44:08.517985 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:08.517908    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:44:40.517898922 +0000 UTC m=+99.061035879 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:44:08.683003 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:08.682920    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:08.783891 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:08.783853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:11.235471 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:11.235433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:11.291165 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:11.291128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:12.337867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:12.337827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pxhn7" event={"ID":"dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb","Type":"ContainerStarted","Data":"1018cb17337b966e9713a6caa74537c02df2c1717f4a31e519b77c771656c852"}
Sep 16 00:44:12.339364 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:12.339338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h6x8j" event={"ID":"eab2796e-f237-4343-a84e-c7393f625e1b","Type":"ContainerStarted","Data":"5ced81560d470470dc55f3716ad316ced7bf88de6c0c4e34bf22989f61b76048"}
Sep 16 00:44:12.339494 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:12.339460    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:44:12.364152 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:12.364095    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h6x8j" podStartSLOduration=65.802905048 podStartE2EDuration="1m10.364074831s" totalImagesPullingTime="4.561169783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:12.363940625 +0000 UTC m=+70.907077601" watchObservedRunningTime="2026-09-16 00:44:12.364074831 +0000 UTC m=+70.907211806"
Sep 16 00:44:12.364537 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:12.364514    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pxhn7" podStartSLOduration=64.500938656 podStartE2EDuration="1m9.364507945s" totalImagesPullingTime="4.863569289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:12.350537203 +0000 UTC m=+70.893674178" watchObservedRunningTime="2026-09-16 00:44:12.364507945 +0000 UTC m=+70.907644916"
Sep 16 00:44:14.445027 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:14.444991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:14.493950 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:14.493918    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:35.188398 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:35.188279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:35.241652 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:35.241625    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:44:40.550734 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:40.550694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:44:40.550734 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:40.550737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:40.550762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:40.550851    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:40.550851    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:40.550928    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:45:44.550912318 +0000 UTC m=+163.094049277 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:40.550857    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:40.550993    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:45:44.550982642 +0000 UTC m=+163.094119595 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:40.550862    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:44:40.551283 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:44:40.551021    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:45:44.551015513 +0000 UTC m=+163.094152465 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:44:43.344730 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:44:43.344697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h6x8j"
Sep 16 00:45:04.702153 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:04.702118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:45:04.802103 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:04.802072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:45:10.766553 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:10.766500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:45:10.766982 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:10.766638    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 00:45:10.766982 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:10.766701    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:12.766687068 +0000 UTC m=+251.309824021 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : secret "metrics-daemon-secret" not found
Sep 16 00:45:39.723890 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:39.723854    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf" podUID="cd6a62e3-f986-4971-8642-42861233c1b6"
Sep 16 00:45:39.733073 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:39.733039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a"
Sep 16 00:45:39.741201 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:39.741177    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hs5c9" podUID="2466104a-6c7b-4788-b0fb-0a7d6144e3d4"
Sep 16 00:45:40.545732 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:40.545700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:45:40.545908 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:40.545700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:45:40.545908 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:40.545700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:45:40.946096 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:40.946006    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:45:44.598006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:44.597944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:45:44.598006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:44.598014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:44.598053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:44.598097    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:44.598137    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:44.598153    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:44.598168    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dfc67b84-z8qvb: secret "image-registry-tls" not found
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:44.598180    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls podName:2466104a-6c7b-4788-b0fb-0a7d6144e3d4 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:46.598163188 +0000 UTC m=+285.141300167 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls") pod "dns-default-hs5c9" (UID: "2466104a-6c7b-4788-b0fb-0a7d6144e3d4") : secret "dns-default-metrics-tls" not found
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:44.598195    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert podName:cd6a62e3-f986-4971-8642-42861233c1b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:46.598189337 +0000 UTC m=+285.141326291 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j45sf" (UID: "cd6a62e3-f986-4971-8642-42861233c1b6") : secret "networking-console-plugin-cert" not found
Sep 16 00:45:44.598454 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:45:44.598221    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls podName:339f2dfa-f2e9-4037-9bbd-84a53206887a nodeName:}" failed. No retries permitted until 2026-09-16 00:47:46.598202155 +0000 UTC m=+285.141339108 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls") pod "image-registry-78dfc67b84-z8qvb" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a") : secret "image-registry-tls" not found
Sep 16 00:45:51.935916 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:45:51.935873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:46:48.106235 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:46:48.106195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:12.813523 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:12.813461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:47:12.813984 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:12.813601    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 00:47:12.813984 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:12.813679    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs podName:0d9cbade-9182-42ac-8e70-5fd8626eb8f3 nodeName:}" failed. No retries permitted until 2026-09-16 00:49:14.813662231 +0000 UTC m=+373.356799185 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs") pod "network-metrics-daemon-mftnx" (UID: "0d9cbade-9182-42ac-8e70-5fd8626eb8f3") : secret "metrics-daemon-secret" not found
Sep 16 00:47:15.850220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.850171    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"]
Sep 16 00:47:15.851943 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.851927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:15.854433 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.854404    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 00:47:15.854559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.854445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.855553 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.855533    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 00:47:15.855672 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.855652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w5rpm\""
Sep 16 00:47:15.857378 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.857361    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.876132 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.876105    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"]
Sep 16 00:47:15.933559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.933523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/485ae4b4-6055-4452-83ce-e1b618a2d23e-config\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:15.933754 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.933569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/485ae4b4-6055-4452-83ce-e1b618a2d23e-serving-cert\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:15.933754 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.933662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmghp\" (UniqueName: \"kubernetes.io/projected/485ae4b4-6055-4452-83ce-e1b618a2d23e-kube-api-access-wmghp\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:15.937950 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.937921    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-c4jbm"]
Sep 16 00:47:15.939639 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.939622    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-q2972"]
Sep 16 00:47:15.939760 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.939743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:15.941433 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.941416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:15.942292 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.942275    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.942699 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.942673    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.942699 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.942700    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-b2rgm\""
Sep 16 00:47:15.942873 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.942709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 00:47:15.943016 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.942999    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 00:47:15.943231 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.943216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 00:47:15.943359 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.943344    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 00:47:15.943671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.943651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.943769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.943651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.943937 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.943919    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xr846\""
Sep 16 00:47:15.949509 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.949458    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-q2972"]
Sep 16 00:47:15.950380 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.950360    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 00:47:15.951782 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.951761    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 00:47:15.952582 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:15.952562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-c4jbm"]
Sep 16 00:47:16.034353 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/485ae4b4-6055-4452-83ce-e1b618a2d23e-serving-cert\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:16.034353 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmxr\" (UniqueName: \"kubernetes.io/projected/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-kube-api-access-zrmxr\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.034598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a63383b4-49c8-447e-982f-7cbf0fa0df04-snapshots\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.034598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a63383b4-49c8-447e-982f-7cbf0fa0df04-trusted-ca-bundle\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.034598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmghp\" (UniqueName: \"kubernetes.io/projected/485ae4b4-6055-4452-83ce-e1b618a2d23e-kube-api-access-wmghp\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:16.034598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a63383b4-49c8-447e-982f-7cbf0fa0df04-service-ca-bundle\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.034598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a63383b4-49c8-447e-982f-7cbf0fa0df04-serving-cert\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.034799 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/485ae4b4-6055-4452-83ce-e1b618a2d23e-config\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:16.034799 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-config\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.034799 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-serving-cert\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.034799 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-trusted-ca\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.034935 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a63383b4-49c8-447e-982f-7cbf0fa0df04-tmp\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.034935 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.034850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkjl\" (UniqueName: \"kubernetes.io/projected/a63383b4-49c8-447e-982f-7cbf0fa0df04-kube-api-access-xdkjl\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.035138 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.035121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/485ae4b4-6055-4452-83ce-e1b618a2d23e-config\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:16.036645 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.036626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/485ae4b4-6055-4452-83ce-e1b618a2d23e-serving-cert\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:16.041797 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.041776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmghp\" (UniqueName: \"kubernetes.io/projected/485ae4b4-6055-4452-83ce-e1b618a2d23e-kube-api-access-wmghp\") pod \"service-ca-operator-cf7c776b7-hwsml\" (UID: \"485ae4b4-6055-4452-83ce-e1b618a2d23e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:16.090834 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.090798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:16.135538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-config\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.135538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-serving-cert\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.135656 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-trusted-ca\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.135656 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a63383b4-49c8-447e-982f-7cbf0fa0df04-tmp\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.135656 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjl\" (UniqueName: \"kubernetes.io/projected/a63383b4-49c8-447e-982f-7cbf0fa0df04-kube-api-access-xdkjl\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.135656 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmxr\" (UniqueName: \"kubernetes.io/projected/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-kube-api-access-zrmxr\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.135834 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a63383b4-49c8-447e-982f-7cbf0fa0df04-snapshots\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.135834 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a63383b4-49c8-447e-982f-7cbf0fa0df04-trusted-ca-bundle\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.135946 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a63383b4-49c8-447e-982f-7cbf0fa0df04-service-ca-bundle\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.135946 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.135903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a63383b4-49c8-447e-982f-7cbf0fa0df04-serving-cert\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.136059 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.136006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a63383b4-49c8-447e-982f-7cbf0fa0df04-tmp\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.136271 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.136242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-config\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.136464 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.136442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a63383b4-49c8-447e-982f-7cbf0fa0df04-service-ca-bundle\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.136729 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.136709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a63383b4-49c8-447e-982f-7cbf0fa0df04-snapshots\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.136974 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.136954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-trusted-ca\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.137116 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.137094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a63383b4-49c8-447e-982f-7cbf0fa0df04-trusted-ca-bundle\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.138131 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.138111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a63383b4-49c8-447e-982f-7cbf0fa0df04-serving-cert\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.138268 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.138248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-serving-cert\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.145837 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.145808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkjl\" (UniqueName: \"kubernetes.io/projected/a63383b4-49c8-447e-982f-7cbf0fa0df04-kube-api-access-xdkjl\") pod \"insights-operator-5844b856db-q2972\" (UID: \"a63383b4-49c8-447e-982f-7cbf0fa0df04\") " pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.145944 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.145922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmxr\" (UniqueName: \"kubernetes.io/projected/0187d13e-b0d9-40f1-91d1-0aea6ba2be15-kube-api-access-zrmxr\") pod \"console-operator-696884fb9c-c4jbm\" (UID: \"0187d13e-b0d9-40f1-91d1-0aea6ba2be15\") " pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.160592 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.160571    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"
Sep 16 00:47:16.190398 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.190363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:16.251281 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.251247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:16.258082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.258059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-q2972"
Sep 16 00:47:16.423916 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.423829    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml"]
Sep 16 00:47:16.426602 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:16.426572    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485ae4b4_6055_4452_83ce_e1b618a2d23e.slice/crio-2f63a9ae716b9320ca9542d0e59ea70ba8a1ab12329f53603afce8bd860e9690 WatchSource:0}: Error finding container 2f63a9ae716b9320ca9542d0e59ea70ba8a1ab12329f53603afce8bd860e9690: Status 404 returned error can't find the container with id 2f63a9ae716b9320ca9542d0e59ea70ba8a1ab12329f53603afce8bd860e9690
Sep 16 00:47:16.519577 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.519545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-c4jbm"]
Sep 16 00:47:16.522692 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:16.522660    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0187d13e_b0d9_40f1_91d1_0aea6ba2be15.slice/crio-e828ff4e22dff0a6df0f9eea89debd87b6f1af34914143f99edd92a30a255e40 WatchSource:0}: Error finding container e828ff4e22dff0a6df0f9eea89debd87b6f1af34914143f99edd92a30a255e40: Status 404 returned error can't find the container with id e828ff4e22dff0a6df0f9eea89debd87b6f1af34914143f99edd92a30a255e40
Sep 16 00:47:16.527320 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.527296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-q2972"]
Sep 16 00:47:16.529962 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:16.529935    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63383b4_49c8_447e_982f_7cbf0fa0df04.slice/crio-6843143772b42b4f0472bd97f942ef8e8e3c88c6ed3de53420e9c8c2b3507d7c WatchSource:0}: Error finding container 6843143772b42b4f0472bd97f942ef8e8e3c88c6ed3de53420e9c8c2b3507d7c: Status 404 returned error can't find the container with id 6843143772b42b4f0472bd97f942ef8e8e3c88c6ed3de53420e9c8c2b3507d7c
Sep 16 00:47:16.779869 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.779832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml" event={"ID":"485ae4b4-6055-4452-83ce-e1b618a2d23e","Type":"ContainerStarted","Data":"2f63a9ae716b9320ca9542d0e59ea70ba8a1ab12329f53603afce8bd860e9690"}
Sep 16 00:47:16.780907 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.780885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" event={"ID":"0187d13e-b0d9-40f1-91d1-0aea6ba2be15","Type":"ContainerStarted","Data":"e828ff4e22dff0a6df0f9eea89debd87b6f1af34914143f99edd92a30a255e40"}
Sep 16 00:47:16.781839 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:16.781821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-q2972" event={"ID":"a63383b4-49c8-447e-982f-7cbf0fa0df04","Type":"ContainerStarted","Data":"6843143772b42b4f0472bd97f942ef8e8e3c88c6ed3de53420e9c8c2b3507d7c"}
Sep 16 00:47:18.638726 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:18.638688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:19.655976 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.655949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:19.658828 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.658800    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"]
Sep 16 00:47:19.662380 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.662363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"
Sep 16 00:47:19.664628 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.664605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 00:47:19.665127 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.665107    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 00:47:19.665220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.665107    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-shlp4\""
Sep 16 00:47:19.672881 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.672857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"]
Sep 16 00:47:19.748542 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.748503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:19.767853 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.767824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kr4\" (UniqueName: \"kubernetes.io/projected/11cdd715-6299-4fa9-942d-064115a5829f-kube-api-access-d2kr4\") pod \"migrator-8b4d697c5-pmsdb\" (UID: \"11cdd715-6299-4fa9-942d-064115a5829f\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"
Sep 16 00:47:19.849334 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.849302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:19.868937 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.868906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kr4\" (UniqueName: \"kubernetes.io/projected/11cdd715-6299-4fa9-942d-064115a5829f-kube-api-access-d2kr4\") pod \"migrator-8b4d697c5-pmsdb\" (UID: \"11cdd715-6299-4fa9-942d-064115a5829f\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"
Sep 16 00:47:19.876982 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:19.876932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kr4\" (UniqueName: \"kubernetes.io/projected/11cdd715-6299-4fa9-942d-064115a5829f-kube-api-access-d2kr4\") pod \"migrator-8b4d697c5-pmsdb\" (UID: \"11cdd715-6299-4fa9-942d-064115a5829f\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"
Sep 16 00:47:20.084214 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.084174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"
Sep 16 00:47:20.355315 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.355280    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb"]
Sep 16 00:47:20.358341 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:20.358317    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cdd715_6299_4fa9_942d_064115a5829f.slice/crio-15dd4be0e56767ab037ee4dd632850d3132f8a382b3f9d10bdc0845699672aae WatchSource:0}: Error finding container 15dd4be0e56767ab037ee4dd632850d3132f8a382b3f9d10bdc0845699672aae: Status 404 returned error can't find the container with id 15dd4be0e56767ab037ee4dd632850d3132f8a382b3f9d10bdc0845699672aae
Sep 16 00:47:20.746436 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.746404    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:20.795182 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.795147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml" event={"ID":"485ae4b4-6055-4452-83ce-e1b618a2d23e","Type":"ContainerStarted","Data":"1045793e99d2aae8521ca8e3c91e18db14ac781fda2b2ce13254ac7dd1f3bf74"}
Sep 16 00:47:20.796886 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.796866    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/0.log"
Sep 16 00:47:20.797007 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.796900    2581 generic.go:356] "Generic (PLEG): container finished" podID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15" containerID="8eac8d55833096ace6d0cd2185db0b83418ec1a8ef144693ca93f0e811a6d513" exitCode=255
Sep 16 00:47:20.797007 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.796958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" event={"ID":"0187d13e-b0d9-40f1-91d1-0aea6ba2be15","Type":"ContainerDied","Data":"8eac8d55833096ace6d0cd2185db0b83418ec1a8ef144693ca93f0e811a6d513"}
Sep 16 00:47:20.797198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.797178    2581 scope.go:117] "RemoveContainer" containerID="8eac8d55833096ace6d0cd2185db0b83418ec1a8ef144693ca93f0e811a6d513"
Sep 16 00:47:20.798287 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.798260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb" event={"ID":"11cdd715-6299-4fa9-942d-064115a5829f","Type":"ContainerStarted","Data":"15dd4be0e56767ab037ee4dd632850d3132f8a382b3f9d10bdc0845699672aae"}
Sep 16 00:47:20.799868 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.799847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-q2972" event={"ID":"a63383b4-49c8-447e-982f-7cbf0fa0df04","Type":"ContainerStarted","Data":"4ff6845d8f61b95c4abc0a08de22575af483e00b808ab5025cea05bdbaaba081"}
Sep 16 00:47:20.810363 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.810309    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml" podStartSLOduration=2.583686588 podStartE2EDuration="5.810286515s" totalImagesPullingTime="3.226599927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:20.809361681 +0000 UTC m=+259.352498659" watchObservedRunningTime="2026-09-16 00:47:20.810286515 +0000 UTC m=+259.353423490"
Sep 16 00:47:20.813282 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.813253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:20.842202 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:20.842128    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-q2972" podStartSLOduration=2.725674745 podStartE2EDuration="5.842104484s" totalImagesPullingTime="3.116429739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:20.841068439 +0000 UTC m=+259.384205413" watchObservedRunningTime="2026-09-16 00:47:20.842104484 +0000 UTC m=+259.385241460"
Sep 16 00:47:21.187634 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.187600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:21.249491 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.249442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:21.373843 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.372631    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fdzml"]
Sep 16 00:47:21.376340 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.376316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.379079 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.379058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 00:47:21.379205 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.379136    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lt8cw\""
Sep 16 00:47:21.379867 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.379846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 00:47:21.386252 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.386194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fdzml"]
Sep 16 00:47:21.389918 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.389893    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:21.484331 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.484239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-crio-socket\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.484331 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.484283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dffvs\" (UniqueName: \"kubernetes.io/projected/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-api-access-dffvs\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.484331 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.484322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.484656 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.484350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-data-volume\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.484656 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.484402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.536286 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.536250    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:21.584904 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.584860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-crio-socket\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.584904 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.584888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dffvs\" (UniqueName: \"kubernetes.io/projected/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-api-access-dffvs\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.584979 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.584910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.584979 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:21.584971    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:21.585041 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.584980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-crio-socket\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.585041 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:21.585036    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:22.085018275 +0000 UTC m=+260.628155243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:21.585139 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.585049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-data-volume\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.585139 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.585089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.585246 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:21.585231    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:21.585297 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:21.585283    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:22.085270197 +0000 UTC m=+260.628407150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:21.585353 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.585338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-data-volume\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.593128 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.593099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dffvs\" (UniqueName: \"kubernetes.io/projected/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-api-access-dffvs\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:21.636818 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.636790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:21.804638 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.804609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/1.log"
Sep 16 00:47:21.805209 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.805016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/0.log"
Sep 16 00:47:21.805209 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.805045    2581 generic.go:356] "Generic (PLEG): container finished" podID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15" containerID="79314103bc09be2b9994e4dba3342074a3ffbb4976a63546ebd829c70fe704ce" exitCode=255
Sep 16 00:47:21.805209 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.805106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" event={"ID":"0187d13e-b0d9-40f1-91d1-0aea6ba2be15","Type":"ContainerDied","Data":"79314103bc09be2b9994e4dba3342074a3ffbb4976a63546ebd829c70fe704ce"}
Sep 16 00:47:21.805209 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.805191    2581 scope.go:117] "RemoveContainer" containerID="8eac8d55833096ace6d0cd2185db0b83418ec1a8ef144693ca93f0e811a6d513"
Sep 16 00:47:21.805414 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.805404    2581 scope.go:117] "RemoveContainer" containerID="79314103bc09be2b9994e4dba3342074a3ffbb4976a63546ebd829c70fe704ce"
Sep 16 00:47:21.805628 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:21.805604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-c4jbm_openshift-console-operator(0187d13e-b0d9-40f1-91d1-0aea6ba2be15)\"" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" podUID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15"
Sep 16 00:47:21.807022 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.806999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb" event={"ID":"11cdd715-6299-4fa9-942d-064115a5829f","Type":"ContainerStarted","Data":"8413ce0029012e8b9718ac3156ddca52232a652d66e200514fef52c3b9305d5b"}
Sep 16 00:47:21.807098 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.807052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb" event={"ID":"11cdd715-6299-4fa9-942d-064115a5829f","Type":"ContainerStarted","Data":"1c1d31302562817a982f9292c8680eea7fcd19561a20811ee390be169ce9afc7"}
Sep 16 00:47:21.833278 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.833221    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pmsdb" podStartSLOduration=1.618056291 podStartE2EDuration="2.833204636s" totalImagesPullingTime="1.215148345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:21.832519577 +0000 UTC m=+260.375656553" watchObservedRunningTime="2026-09-16 00:47:21.833204636 +0000 UTC m=+260.376341611"
Sep 16 00:47:21.907873 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:21.907840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:22.003799 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:22.003763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:22.089717 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:22.089619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:22.089717 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:22.089694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:22.089907 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:22.089752    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:22.089907 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:22.089829    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:22.089907 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:22.089834    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.089811631 +0000 UTC m=+261.632948590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:22.089907 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:22.089901    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.089882502 +0000 UTC m=+261.633019458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:22.811624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:22.811596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/1.log"
Sep 16 00:47:22.812064 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:22.811913    2581 scope.go:117] "RemoveContainer" containerID="79314103bc09be2b9994e4dba3342074a3ffbb4976a63546ebd829c70fe704ce"
Sep 16 00:47:22.812064 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:22.812055    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-c4jbm_openshift-console-operator(0187d13e-b0d9-40f1-91d1-0aea6ba2be15)\"" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" podUID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15"
Sep 16 00:47:23.048240 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.048206    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9zsqj"]
Sep 16 00:47:23.052223 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.052207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.054536 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.054515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 00:47:23.054674 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.054582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 00:47:23.054738 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.054669    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 00:47:23.054738 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.054701    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kss2d\""
Sep 16 00:47:23.055374 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.055357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 00:47:23.057459 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.057436    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9zsqj"]
Sep 16 00:47:23.080167 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.080097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:23.100868 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.100837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56qb\" (UniqueName: \"kubernetes.io/projected/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-kube-api-access-r56qb\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.100979 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.100889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-signing-key\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.100979 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.100936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:23.101051 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.100997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:23.101051 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.101022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-signing-cabundle\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.101110 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:23.101048    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:23.101110 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:23.101104    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:25.101088452 +0000 UTC m=+263.644225405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:23.101180 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:23.101129    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:23.101180 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:23.101170    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:25.101155571 +0000 UTC m=+263.644292525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:23.180937 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.180899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:23.202046 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.202010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56qb\" (UniqueName: \"kubernetes.io/projected/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-kube-api-access-r56qb\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.202181 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.202074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-signing-key\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.202181 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.202157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-signing-cabundle\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.202796 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.202775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-signing-cabundle\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.204666 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.204635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-signing-key\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.210304 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.210277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56qb\" (UniqueName: \"kubernetes.io/projected/bd0a7028-80c6-40cf-9760-dd67a4a4c03b-kube-api-access-r56qb\") pod \"service-ca-568869fdc5-9zsqj\" (UID: \"bd0a7028-80c6-40cf-9760-dd67a4a4c03b\") " pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.361257 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.361176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9zsqj"
Sep 16 00:47:23.631917 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.631837    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9zsqj"]
Sep 16 00:47:23.634396 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:23.634365    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0a7028_80c6_40cf_9760_dd67a4a4c03b.slice/crio-eb545560259b1d43de866976fe7e910ecfc77055055aa8010f796d9a5fea3a4c WatchSource:0}: Error finding container eb545560259b1d43de866976fe7e910ecfc77055055aa8010f796d9a5fea3a4c: Status 404 returned error can't find the container with id eb545560259b1d43de866976fe7e910ecfc77055055aa8010f796d9a5fea3a4c
Sep 16 00:47:23.816453 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.816403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9zsqj" event={"ID":"bd0a7028-80c6-40cf-9760-dd67a4a4c03b","Type":"ContainerStarted","Data":"20af088018036aea568cbac98639cbd46b371f043c9427581ab8b87eb6714669"}
Sep 16 00:47:23.816877 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.816471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9zsqj" event={"ID":"bd0a7028-80c6-40cf-9760-dd67a4a4c03b","Type":"ContainerStarted","Data":"eb545560259b1d43de866976fe7e910ecfc77055055aa8010f796d9a5fea3a4c"}
Sep 16 00:47:23.837342 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.837285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-9zsqj" podStartSLOduration=0.837268819 podStartE2EDuration="837.268819ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:23.8366618 +0000 UTC m=+262.379798774" watchObservedRunningTime="2026-09-16 00:47:23.837268819 +0000 UTC m=+262.380405793"
Sep 16 00:47:23.936949 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.936837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:23.941207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:23.941181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-46zrq_6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405/dns-node-resolver/0.log"
Sep 16 00:47:24.665554 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:24.665521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:24.822799 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:24.822754    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:24.941587 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:24.941499    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8z8zz_d8b1989d-a634-4622-82c7-9ba42c0b829b/node-ca/0.log"
Sep 16 00:47:24.942032 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:24.942008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:25.043700 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:25.043659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:25.120728 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:25.120685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:25.120912 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:25.120774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:25.120912 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:25.120821    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:25.120912 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:25.120890    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:25.121026 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:25.120914    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:29.120891481 +0000 UTC m=+267.664028441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:25.121026 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:25.120941    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:29.120925179 +0000 UTC m=+267.664062138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:25.155390 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:25.155355    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:25.562013 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:25.561976    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:25.742162 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:25.742132    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cj78c_45b30463-b1d8-4f57-8f94-80411105dd84/serve-healthcheck-canary/0.log"
Sep 16 00:47:25.957598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:25.957511    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:26.252456 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:26.252370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:26.252456 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:26.252438    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:26.252812 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:26.252798    2581 scope.go:117] "RemoveContainer" containerID="79314103bc09be2b9994e4dba3342074a3ffbb4976a63546ebd829c70fe704ce"
Sep 16 00:47:26.252960 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:26.252942    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-c4jbm_openshift-console-operator(0187d13e-b0d9-40f1-91d1-0aea6ba2be15)\"" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" podUID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15"
Sep 16 00:47:26.343178 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:26.343145    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pmsdb_11cdd715-6299-4fa9-942d-064115a5829f/migrator/0.log"
Sep 16 00:47:26.480896 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:26.480862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:26.542536 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:26.542508    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pmsdb_11cdd715-6299-4fa9-942d-064115a5829f/graceful-termination/0.log"
Sep 16 00:47:26.582400 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:26.582363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:27.158671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:27.158635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:27.364147 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:27.364110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:27.561297 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:27.561261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:27.761943 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:27.761910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:28.529007 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:28.528974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:28.582247 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:28.582205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:28.956520 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:28.956425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:29.157041 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:29.157005    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:29.158052 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:29.158032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:29.158143 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:29.158110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:29.158203 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:29.158171    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:29.158203 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:29.158172    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:29.158309 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:29.158226    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:37.158214504 +0000 UTC m=+275.701351458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:29.158309 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:29.158239    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls podName:faa5c9f6-fd29-4380-a883-42cbb3b78c2f nodeName:}" failed. No retries permitted until 2026-09-16 00:47:37.15823357 +0000 UTC m=+275.701370523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdzml" (UID: "faa5c9f6-fd29-4380-a883-42cbb3b78c2f") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:29.365245 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:29.365212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:29.563269 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:29.563228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:30.926989 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:30.926956    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:31.026621 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:31.026579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:31.131690 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:31.131660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:31.361161 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:31.361125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:31.558126 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:31.558080    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:31.959744 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:31.959712    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:32.560077 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:32.560046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:32.887400 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:32.887318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:32.987565 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:32.987536    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:33.161616 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:33.161534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:33.364366 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:33.364336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:34.359993 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:34.359958    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:34.569489 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:34.569445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:34.637304 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:34.637219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:34.758079 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:34.758047    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:34.965940 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:34.965857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:35.160926 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:35.160891    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:35.355535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:35.355474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:35.759725 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:35.759646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:37.228039 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.228006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:37.228422 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.228058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:37.228592 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.228570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:37.230325 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.230306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/faa5c9f6-fd29-4380-a883-42cbb3b78c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdzml\" (UID: \"faa5c9f6-fd29-4380-a883-42cbb3b78c2f\") " pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:37.287882 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.287843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fdzml"
Sep 16 00:47:37.558271 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.558236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fdzml"]
Sep 16 00:47:37.562345 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:37.562314    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa5c9f6_fd29_4380_a883_42cbb3b78c2f.slice/crio-742dd0ab446755124b9c03f22ba6a3b9d4d4b9db2009e019c4fe41a54f71af83 WatchSource:0}: Error finding container 742dd0ab446755124b9c03f22ba6a3b9d4d4b9db2009e019c4fe41a54f71af83: Status 404 returned error can't find the container with id 742dd0ab446755124b9c03f22ba6a3b9d4d4b9db2009e019c4fe41a54f71af83
Sep 16 00:47:37.866986 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.866903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdzml" event={"ID":"faa5c9f6-fd29-4380-a883-42cbb3b78c2f","Type":"ContainerStarted","Data":"410651b3501fef0e3493f55deae7207e7585915e649fc03cf88001e0bbecf65c"}
Sep 16 00:47:37.866986 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.866960    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdzml" event={"ID":"faa5c9f6-fd29-4380-a883-42cbb3b78c2f","Type":"ContainerStarted","Data":"742dd0ab446755124b9c03f22ba6a3b9d4d4b9db2009e019c4fe41a54f71af83"}
Sep 16 00:47:37.935738 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:37.935701    2581 scope.go:117] "RemoveContainer" containerID="79314103bc09be2b9994e4dba3342074a3ffbb4976a63546ebd829c70fe704ce"
Sep 16 00:47:38.871380 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:38.871353    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:47:38.871830 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:38.871813    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/1.log"
Sep 16 00:47:38.871917 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:38.871855    2581 generic.go:356] "Generic (PLEG): container finished" podID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15" containerID="4d4d1fc189dcb12f0afe0cb1b169d31638185fd24e948ac538439835bcbbe573" exitCode=255
Sep 16 00:47:38.871965 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:38.871930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" event={"ID":"0187d13e-b0d9-40f1-91d1-0aea6ba2be15","Type":"ContainerDied","Data":"4d4d1fc189dcb12f0afe0cb1b169d31638185fd24e948ac538439835bcbbe573"}
Sep 16 00:47:38.872010 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:38.871988    2581 scope.go:117] "RemoveContainer" containerID="79314103bc09be2b9994e4dba3342074a3ffbb4976a63546ebd829c70fe704ce"
Sep 16 00:47:38.872270 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:38.872254    2581 scope.go:117] "RemoveContainer" containerID="4d4d1fc189dcb12f0afe0cb1b169d31638185fd24e948ac538439835bcbbe573"
Sep 16 00:47:38.872457 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:38.872430    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-c4jbm_openshift-console-operator(0187d13e-b0d9-40f1-91d1-0aea6ba2be15)\"" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" podUID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15"
Sep 16 00:47:38.873733 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:38.873710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdzml" event={"ID":"faa5c9f6-fd29-4380-a883-42cbb3b78c2f","Type":"ContainerStarted","Data":"8b948d305aba36ac0c2e82f7a6d3fe1d124090f5db2830bf8bc8fc48bbfa197c"}
Sep 16 00:47:39.878017 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:39.877987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:47:39.880166 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:39.880139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdzml" event={"ID":"faa5c9f6-fd29-4380-a883-42cbb3b78c2f","Type":"ContainerStarted","Data":"1c64ea94df89810c510f3d49a31786fd1a7bc39f8c2f0045649b45c2c6742e44"}
Sep 16 00:47:39.896266 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:39.896221    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fdzml" podStartSLOduration=16.872305175 podStartE2EDuration="18.896203723s" totalImagesPullingTime="2.023898548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:47:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:39.895068468 +0000 UTC m=+278.438205446" watchObservedRunningTime="2026-09-16 00:47:39.896203723 +0000 UTC m=+278.439340698"
Sep 16 00:47:40.615525 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:40.615474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:40.669421 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:40.669391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:42.379691 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:42.379658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:42.478985 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:42.478940    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:43.546731 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:43.546682    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf" podUID="cd6a62e3-f986-4971-8642-42861233c1b6"
Sep 16 00:47:43.546731 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:43.546699    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a"
Sep 16 00:47:43.546731 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:43.546699    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hs5c9" podUID="2466104a-6c7b-4788-b0fb-0a7d6144e3d4"
Sep 16 00:47:43.892922 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:43.892844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:47:43.893064 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:43.892854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:47:43.893064 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:43.892854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:47:44.803777 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.803738    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:44.855176 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.855143    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"]
Sep 16 00:47:44.858141 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.858125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:44.860323 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.860291    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 00:47:44.860433 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.860324    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 00:47:44.861259 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.861241    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 00:47:44.861335 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.861282    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qtclq\""
Sep 16 00:47:44.861335 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.861302    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 00:47:44.861335 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.861323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 00:47:44.865206 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.865185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"]
Sep 16 00:47:44.867364 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.867344    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:44.993727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.993683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:44.993948 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.993741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6zt\" (UniqueName: \"kubernetes.io/projected/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-kube-api-access-tp6zt\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:44.993948 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.993794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:44.993948 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:44.993857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.049689 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.049658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:45.094897 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.094804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.094897 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.094859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.095103 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.094946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.095103 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.094980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6zt\" (UniqueName: \"kubernetes.io/projected/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-kube-api-access-tp6zt\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.095572 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.095551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.097315 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.097284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.097315 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.097303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.102379 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.102344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6zt\" (UniqueName: \"kubernetes.io/projected/1a3b3e9c-e324-48e2-8302-6ff24fbf9463-kube-api-access-tp6zt\") pod \"prometheus-operator-68bcd7f987-rnd5b\" (UID: \"1a3b3e9c-e324-48e2-8302-6ff24fbf9463\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.167837 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.167804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"
Sep 16 00:47:45.432276 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.432164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b"]
Sep 16 00:47:45.436267 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:45.436236    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3b3e9c_e324_48e2_8302_6ff24fbf9463.slice/crio-830563c8b8c2d5ae09136658e07c44c8ea1d70b1ac35d8ec0d12ba57f0a82cf4 WatchSource:0}: Error finding container 830563c8b8c2d5ae09136658e07c44c8ea1d70b1ac35d8ec0d12ba57f0a82cf4: Status 404 returned error can't find the container with id 830563c8b8c2d5ae09136658e07c44c8ea1d70b1ac35d8ec0d12ba57f0a82cf4
Sep 16 00:47:45.900075 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:45.900037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b" event={"ID":"1a3b3e9c-e324-48e2-8302-6ff24fbf9463","Type":"ContainerStarted","Data":"830563c8b8c2d5ae09136658e07c44c8ea1d70b1ac35d8ec0d12ba57f0a82cf4"}
Sep 16 00:47:46.252121 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.252033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:46.252121 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.252099    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:47:46.252452 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.252435    2581 scope.go:117] "RemoveContainer" containerID="4d4d1fc189dcb12f0afe0cb1b169d31638185fd24e948ac538439835bcbbe573"
Sep 16 00:47:46.252654 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:46.252635    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-c4jbm_openshift-console-operator(0187d13e-b0d9-40f1-91d1-0aea6ba2be15)\"" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" podUID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15"
Sep 16 00:47:46.608759 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.608721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:47:46.609000 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.608788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:47:46.609000 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.608847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:47:46.611558 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.611511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2466104a-6c7b-4788-b0fb-0a7d6144e3d4-metrics-tls\") pod \"dns-default-hs5c9\" (UID: \"2466104a-6c7b-4788-b0fb-0a7d6144e3d4\") " pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:47:46.611704 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.611651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"image-registry-78dfc67b84-z8qvb\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") " pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:47:46.611704 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.611674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd6a62e3-f986-4971-8642-42861233c1b6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j45sf\" (UID: \"cd6a62e3-f986-4971-8642-42861233c1b6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:47:46.896463 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.896437    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n5l29\""
Sep 16 00:47:46.896584 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.896438    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l2glv\""
Sep 16 00:47:46.896584 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.896502    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gnmcb\""
Sep 16 00:47:46.903759 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.903706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:47:46.903759 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.903734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:47:46.904180 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.903706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"
Sep 16 00:47:46.907350 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.907320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b" event={"ID":"1a3b3e9c-e324-48e2-8302-6ff24fbf9463","Type":"ContainerStarted","Data":"b2a61ae0b3bf2a9c7ef66e0f31d4d414c31df59e92035623a58573af11f4926f"}
Sep 16 00:47:46.907446 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.907428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b" event={"ID":"1a3b3e9c-e324-48e2-8302-6ff24fbf9463","Type":"ContainerStarted","Data":"668daf10f894975f3ca218c8d18b1db779a34e4cfb6588583406ff13681faa59"}
Sep 16 00:47:46.924005 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:46.923896    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rnd5b" podStartSLOduration=1.566527045 podStartE2EDuration="2.923874039s" totalImagesPullingTime="1.357346994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:46.921808766 +0000 UTC m=+285.464945743" watchObservedRunningTime="2026-09-16 00:47:46.923874039 +0000 UTC m=+285.467011014"
Sep 16 00:47:47.055556 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.055520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:47.185005 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.184922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78dfc67b84-z8qvb"]
Sep 16 00:47:47.187800 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:47.187773    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339f2dfa_f2e9_4037_9bbd_84a53206887a.slice/crio-8219f1e91a7e984b1ba9c8cb3e3ff8a0360995be0134f39caa1957b94ce8e221 WatchSource:0}: Error finding container 8219f1e91a7e984b1ba9c8cb3e3ff8a0360995be0134f39caa1957b94ce8e221: Status 404 returned error can't find the container with id 8219f1e91a7e984b1ba9c8cb3e3ff8a0360995be0134f39caa1957b94ce8e221
Sep 16 00:47:47.192745 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.192717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hs5c9"]
Sep 16 00:47:47.195718 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:47.195689    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2466104a_6c7b_4788_b0fb_0a7d6144e3d4.slice/crio-1ca287cca064d00bed3c29652fd52ba26617c08873eba1f11011afb013c8ca6b WatchSource:0}: Error finding container 1ca287cca064d00bed3c29652fd52ba26617c08873eba1f11011afb013c8ca6b: Status 404 returned error can't find the container with id 1ca287cca064d00bed3c29652fd52ba26617c08873eba1f11011afb013c8ca6b
Sep 16 00:47:47.203862 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.203842    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf"]
Sep 16 00:47:47.207116 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:47.207091    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6a62e3_f986_4971_8642_42861233c1b6.slice/crio-aaa2baf6d86357eff5efebbb8a5e0edcab6afd72934ae1ef27d82cbc721257c7 WatchSource:0}: Error finding container aaa2baf6d86357eff5efebbb8a5e0edcab6afd72934ae1ef27d82cbc721257c7: Status 404 returned error can't find the container with id aaa2baf6d86357eff5efebbb8a5e0edcab6afd72934ae1ef27d82cbc721257c7
Sep 16 00:47:47.750915 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.750883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:47.801907 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.801793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:47.913576 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.913537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" event={"ID":"339f2dfa-f2e9-4037-9bbd-84a53206887a","Type":"ContainerStarted","Data":"440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204"}
Sep 16 00:47:47.914048 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.913600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" event={"ID":"339f2dfa-f2e9-4037-9bbd-84a53206887a","Type":"ContainerStarted","Data":"8219f1e91a7e984b1ba9c8cb3e3ff8a0360995be0134f39caa1957b94ce8e221"}
Sep 16 00:47:47.914048 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.913627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:47:47.915403 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.915370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf" event={"ID":"cd6a62e3-f986-4971-8642-42861233c1b6","Type":"ContainerStarted","Data":"aaa2baf6d86357eff5efebbb8a5e0edcab6afd72934ae1ef27d82cbc721257c7"}
Sep 16 00:47:47.918031 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.917997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hs5c9" event={"ID":"2466104a-6c7b-4788-b0fb-0a7d6144e3d4","Type":"ContainerStarted","Data":"1ca287cca064d00bed3c29652fd52ba26617c08873eba1f11011afb013c8ca6b"}
Sep 16 00:47:47.933968 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:47.933904    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" podStartSLOduration=284.933883165 podStartE2EDuration="4m44.933883165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:47.933110817 +0000 UTC m=+286.476247819" watchObservedRunningTime="2026-09-16 00:47:47.933883165 +0000 UTC m=+286.477020121"
Sep 16 00:47:48.923668 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:48.923621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf" event={"ID":"cd6a62e3-f986-4971-8642-42861233c1b6","Type":"ContainerStarted","Data":"1e2a80cb0f0ad71cc8ad03fd7061c077f1481393f18b940719383503ff27b970"}
Sep 16 00:47:48.944477 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:48.944419    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j45sf" podStartSLOduration=266.769572291 podStartE2EDuration="4m27.944402109s" totalImagesPullingTime="1.174829818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:48.943569472 +0000 UTC m=+287.486706447" watchObservedRunningTime="2026-09-16 00:47:48.944402109 +0000 UTC m=+287.487539083"
Sep 16 00:47:48.952734 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:48.952697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.047579 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.047544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.149089 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.149050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.183342 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.183252    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"]
Sep 16 00:47:49.186370 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.185866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.189507 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.188371    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9c6n2"]
Sep 16 00:47:49.189507 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.188459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 00:47:49.189507 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.188690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 00:47:49.189507 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.188887    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kzgzs\""
Sep 16 00:47:49.190879 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.190858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.193530 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.193206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ldbkq\""
Sep 16 00:47:49.193530 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.193284    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 00:47:49.193530 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.193411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 00:47:49.193730 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.193665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 00:47:49.198914 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.197949    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"]
Sep 16 00:47:49.210019 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.202686    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"]
Sep 16 00:47:49.241986 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.239719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.242233 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.242217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkdf\" (UniqueName: \"kubernetes.io/projected/ed050320-b082-466b-b27a-0ffc70a5f972-kube-api-access-8lkdf\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.242346 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.242329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.242434 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.242421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed050320-b082-466b-b27a-0ffc70a5f972-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.243046 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.243020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"]
Sep 16 00:47:49.243343 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.243273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.245825 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.245808    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 00:47:49.246240 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.246219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 00:47:49.246548 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.246529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qqqqc\""
Sep 16 00:47:49.247256 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.246631    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 00:47:49.250702 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.250679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.343655 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.343611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.343838 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.343747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed050320-b082-466b-b27a-0ffc70a5f972-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.344104 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f93155b-bfb8-4840-a141-bf9418701d29-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.344287 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.344429 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-tls\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.344582 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f93155b-bfb8-4840-a141-bf9418701d29-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.344720 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-root\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.344863 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f16b75b6-03e7-4fdd-9f23-359f953e275b-metrics-client-ca\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.345011 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-textfile\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.345111 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.345086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.345186 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.344997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed050320-b082-466b-b27a-0ffc70a5f972-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.345244 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.345122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlrf\" (UniqueName: \"kubernetes.io/projected/f16b75b6-03e7-4fdd-9f23-359f953e275b-kube-api-access-gnlrf\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.345305 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.345273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-wtmp\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.345364 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.345334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.345421 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.345360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-sys\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.345611 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:49.345570    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 00:47:49.345713 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.345598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.345820 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:49.345705    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-tls podName:ed050320-b082-466b-b27a-0ffc70a5f972 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:49.845684083 +0000 UTC m=+288.388821050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-pzmhb" (UID: "ed050320-b082-466b-b27a-0ffc70a5f972") : secret "openshift-state-metrics-tls" not found
Sep 16 00:47:49.345968 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.345951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.346100 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.346082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhksd\" (UniqueName: \"kubernetes.io/projected/3f93155b-bfb8-4840-a141-bf9418701d29-kube-api-access-dhksd\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.346185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.346156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkdf\" (UniqueName: \"kubernetes.io/projected/ed050320-b082-466b-b27a-0ffc70a5f972-kube-api-access-8lkdf\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.346272 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.346203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.351098 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.351053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.357107 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.357062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkdf\" (UniqueName: \"kubernetes.io/projected/ed050320-b082-466b-b27a-0ffc70a5f972-kube-api-access-8lkdf\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.417441 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.417410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.447265 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-root\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447387 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f16b75b6-03e7-4fdd-9f23-359f953e275b-metrics-client-ca\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447387 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-textfile\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447387 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.447387 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlrf\" (UniqueName: \"kubernetes.io/projected/f16b75b6-03e7-4fdd-9f23-359f953e275b-kube-api-access-gnlrf\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447387 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-root\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447651 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-wtmp\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447651 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-sys\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447651 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447651 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447651 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhksd\" (UniqueName: \"kubernetes.io/projected/3f93155b-bfb8-4840-a141-bf9418701d29-kube-api-access-dhksd\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-textfile\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f93155b-bfb8-4840-a141-bf9418701d29-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-tls\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f93155b-bfb8-4840-a141-bf9418701d29-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-wtmp\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.447945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.447911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16b75b6-03e7-4fdd-9f23-359f953e275b-sys\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.448320 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:49.447992    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 00:47:49.448320 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.448005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f16b75b6-03e7-4fdd-9f23-359f953e275b-metrics-client-ca\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.448320 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:49.448048    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-tls podName:f16b75b6-03e7-4fdd-9f23-359f953e275b nodeName:}" failed. No retries permitted until 2026-09-16 00:47:49.948030193 +0000 UTC m=+288.491167159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-tls") pod "node-exporter-9c6n2" (UID: "f16b75b6-03e7-4fdd-9f23-359f953e275b") : secret "node-exporter-tls" not found
Sep 16 00:47:49.448320 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.448202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f93155b-bfb8-4840-a141-bf9418701d29-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.448538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.448384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.448538 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.448520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f93155b-bfb8-4840-a141-bf9418701d29-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.448910 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.448833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.450662 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.450632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.450799 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.450778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.451156 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.451101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f93155b-bfb8-4840-a141-bf9418701d29-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.455822 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.455799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlrf\" (UniqueName: \"kubernetes.io/projected/f16b75b6-03e7-4fdd-9f23-359f953e275b-kube-api-access-gnlrf\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.456307 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.456265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhksd\" (UniqueName: \"kubernetes.io/projected/3f93155b-bfb8-4840-a141-bf9418701d29-kube-api-access-dhksd\") pod \"kube-state-metrics-59fbdd77d-t87km\" (UID: \"3f93155b-bfb8-4840-a141-bf9418701d29\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.518643 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.518593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.555422 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.555392    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"
Sep 16 00:47:49.621021 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.620992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.831262 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.831229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-t87km"]
Sep 16 00:47:49.834157 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:49.834132    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f93155b_bfb8_4840_a141_bf9418701d29.slice/crio-ddecc8a689b6f16d9b543ad0f44050dd137934e2f274542eedab3a76e976b3ee WatchSource:0}: Error finding container ddecc8a689b6f16d9b543ad0f44050dd137934e2f274542eedab3a76e976b3ee: Status 404 returned error can't find the container with id ddecc8a689b6f16d9b543ad0f44050dd137934e2f274542eedab3a76e976b3ee
Sep 16 00:47:49.851986 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.851961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.854247 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.854227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed050320-b082-466b-b27a-0ffc70a5f972-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pzmhb\" (UID: \"ed050320-b082-466b-b27a-0ffc70a5f972\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:49.928166 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.928125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km" event={"ID":"3f93155b-bfb8-4840-a141-bf9418701d29","Type":"ContainerStarted","Data":"ddecc8a689b6f16d9b543ad0f44050dd137934e2f274542eedab3a76e976b3ee"}
Sep 16 00:47:49.929930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.929904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hs5c9" event={"ID":"2466104a-6c7b-4788-b0fb-0a7d6144e3d4","Type":"ContainerStarted","Data":"2815fe0298d02c2f8599fc0fcce58f44ad2dae19c42d7d6bff1536a8b548b1c6"}
Sep 16 00:47:49.930038 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.929950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hs5c9" event={"ID":"2466104a-6c7b-4788-b0fb-0a7d6144e3d4","Type":"ContainerStarted","Data":"271080ce21231c8cd40b4560038518f783eb7af0b05dd38066e7a3d3c91870b5"}
Sep 16 00:47:49.940472 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.940444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:49.947992 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.947943    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hs5c9" podStartSLOduration=251.70888351 podStartE2EDuration="4m13.94792931s" totalImagesPullingTime="2.2390458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:49.946974117 +0000 UTC m=+288.490111087" watchObservedRunningTime="2026-09-16 00:47:49.94792931 +0000 UTC m=+288.491066284"
Sep 16 00:47:49.952307 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.952286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-tls\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:49.954732 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:49.954712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f16b75b6-03e7-4fdd-9f23-359f953e275b-node-exporter-tls\") pod \"node-exporter-9c6n2\" (UID: \"f16b75b6-03e7-4fdd-9f23-359f953e275b\") " pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:50.113372 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.113280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"
Sep 16 00:47:50.142377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.142343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9c6n2"
Sep 16 00:47:50.157668 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:50.157631    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16b75b6_03e7_4fdd_9f23_359f953e275b.slice/crio-69f65adf21ec8aff28e7c4622e09372fe96510202669790c9b6fd3f8c5ff1585 WatchSource:0}: Error finding container 69f65adf21ec8aff28e7c4622e09372fe96510202669790c9b6fd3f8c5ff1585: Status 404 returned error can't find the container with id 69f65adf21ec8aff28e7c4622e09372fe96510202669790c9b6fd3f8c5ff1585
Sep 16 00:47:50.181227 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.181198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:50.259929 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.259895    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:47:50.263272 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.263245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.264940 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.264914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:50.265296 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.265275    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 00:47:50.265590 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.265529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 00:47:50.265590 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.265538    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 00:47:50.265590 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.265554    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 00:47:50.265809 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.265673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 00:47:50.265809 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.265727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6qrrx\""
Sep 16 00:47:50.265952 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.265934    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 00:47:50.266095 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.266009    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 00:47:50.266183 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.266115    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 00:47:50.266275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.266243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 00:47:50.276527 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.276410    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:47:50.355887 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.355847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356049 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.355894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356049 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.355949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-volume\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356049 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.355997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356049 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-out\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356384 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356384 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356384 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5sf\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-kube-api-access-6m5sf\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356537 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-web-config\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.356537 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.356440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.393015 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.392989    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb"]
Sep 16 00:47:50.395375 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.395342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:50.396144 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:50.396112    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded050320_b082_466b_b27a_0ffc70a5f972.slice/crio-cdd5b83a0114f0897183c3408f328025e539074006ede9e28e451f819c9d4179 WatchSource:0}: Error finding container cdd5b83a0114f0897183c3408f328025e539074006ede9e28e451f819c9d4179: Status 404 returned error can't find the container with id cdd5b83a0114f0897183c3408f328025e539074006ede9e28e451f819c9d4179
Sep 16 00:47:50.457568 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.457751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.457751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.457751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-volume\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.457751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.457751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.458006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.458006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.458006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-out\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.458006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.458006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.458006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5sf\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-kube-api-access-6m5sf\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.458006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.457968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-web-config\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.459279 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.458989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.459279 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.459120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.460559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.460038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.460920 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.460808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.462436 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.461694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.462436 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.462051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.462436 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.462397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.463346 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.463304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-web-config\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.463855 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.463832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.464524 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.464462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.464883 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.464860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-volume\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.465006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.464984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-out\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.467442 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.467420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5sf\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-kube-api-access-6m5sf\") pod \"alertmanager-main-0\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.574675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.574617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:47:50.869140 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.869046    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:47:50.938381 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.938337    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb" event={"ID":"ed050320-b082-466b-b27a-0ffc70a5f972","Type":"ContainerStarted","Data":"640fcad62db0dd445d08299a4e7c63a6c5f8f46607eb7e0b8a6c3b8669d18b75"}
Sep 16 00:47:50.938854 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.938407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb" event={"ID":"ed050320-b082-466b-b27a-0ffc70a5f972","Type":"ContainerStarted","Data":"a88d9ac588b611890d539ed74abc390210f70eb25389bd373cf67134489c1598"}
Sep 16 00:47:50.938854 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.938433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb" event={"ID":"ed050320-b082-466b-b27a-0ffc70a5f972","Type":"ContainerStarted","Data":"cdd5b83a0114f0897183c3408f328025e539074006ede9e28e451f819c9d4179"}
Sep 16 00:47:50.940307 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.940222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c6n2" event={"ID":"f16b75b6-03e7-4fdd-9f23-359f953e275b","Type":"ContainerStarted","Data":"69f65adf21ec8aff28e7c4622e09372fe96510202669790c9b6fd3f8c5ff1585"}
Sep 16 00:47:50.940543 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:50.940472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:47:51.043100 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:51.043067    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba251c2_2caa_42d6_91ed_ac7eaa417a92.slice/crio-0cfd80fa100a7924f731da39c12e9fb5e5d1e57caa88518491d05d3bd871a08b WatchSource:0}: Error finding container 0cfd80fa100a7924f731da39c12e9fb5e5d1e57caa88518491d05d3bd871a08b: Status 404 returned error can't find the container with id 0cfd80fa100a7924f731da39c12e9fb5e5d1e57caa88518491d05d3bd871a08b
Sep 16 00:47:51.946557 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.946519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb" event={"ID":"ed050320-b082-466b-b27a-0ffc70a5f972","Type":"ContainerStarted","Data":"94e950c2a0d8a40e8fcb12676e5c6de8a6ea5afe28b613ba46d6bbac5d867a71"}
Sep 16 00:47:51.948600 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.948571    2581 generic.go:356] "Generic (PLEG): container finished" podID="f16b75b6-03e7-4fdd-9f23-359f953e275b" containerID="9ce3ccae2fc231c13c75be3968a3b1426ea0c5076399cc1720158c38e02ece2f" exitCode=0
Sep 16 00:47:51.948740 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.948688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c6n2" event={"ID":"f16b75b6-03e7-4fdd-9f23-359f953e275b","Type":"ContainerDied","Data":"9ce3ccae2fc231c13c75be3968a3b1426ea0c5076399cc1720158c38e02ece2f"}
Sep 16 00:47:51.951474 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.951449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km" event={"ID":"3f93155b-bfb8-4840-a141-bf9418701d29","Type":"ContainerStarted","Data":"123b6d638720f0b791de0f8aef54f16e8cd29693b19832d4de39b739f85f2cb7"}
Sep 16 00:47:51.951613 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.951520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km" event={"ID":"3f93155b-bfb8-4840-a141-bf9418701d29","Type":"ContainerStarted","Data":"4d2658827b59e9ba221f256e08df4e9cb660f40ff9a065ce2800f314a2ffbb02"}
Sep 16 00:47:51.951613 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.951553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km" event={"ID":"3f93155b-bfb8-4840-a141-bf9418701d29","Type":"ContainerStarted","Data":"0f6879328abd192cb00580100988cc918ffdf1e39bfc5cb1dfd895990eb2d714"}
Sep 16 00:47:51.952896 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.952862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerStarted","Data":"0cfd80fa100a7924f731da39c12e9fb5e5d1e57caa88518491d05d3bd871a08b"}
Sep 16 00:47:51.964019 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.963964    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pzmhb" podStartSLOduration=1.738225898 podStartE2EDuration="2.96394513s" totalImagesPullingTime="1.225719232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:51.961734213 +0000 UTC m=+290.504871188" watchObservedRunningTime="2026-09-16 00:47:51.96394513 +0000 UTC m=+290.507082109"
Sep 16 00:47:51.996216 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:51.996159    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-t87km" podStartSLOduration=1.547935538 podStartE2EDuration="2.99613853s" totalImagesPullingTime="1.448202992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:51.995001336 +0000 UTC m=+290.538138312" watchObservedRunningTime="2026-09-16 00:47:51.99613853 +0000 UTC m=+290.539275507"
Sep 16 00:47:52.136027 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.135983    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:52.235938 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.235904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:52.337831 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.337748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:52.437190 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.437155    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:52.958275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.958175    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerID="cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7" exitCode=0
Sep 16 00:47:52.958275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.958262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7"}
Sep 16 00:47:52.960476 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.960450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c6n2" event={"ID":"f16b75b6-03e7-4fdd-9f23-359f953e275b","Type":"ContainerStarted","Data":"b7fe0c96ca5267e0949a7f3b8688941a164303881f1b104b3ba5d41202e4763e"}
Sep 16 00:47:52.960587 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.960516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c6n2" event={"ID":"f16b75b6-03e7-4fdd-9f23-359f953e275b","Type":"ContainerStarted","Data":"9b846bb40ed06de2693fd3571f39bac28d28e87c05f0aba7cb1332b82d09eebf"}
Sep 16 00:47:53.002074 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:52.998560    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9c6n2" podStartSLOduration=2.783846615 podStartE2EDuration="3.998539214s" totalImagesPullingTime="1.126978708s" totalInitContainerRuntime="87.713891ms" isStatefulPod=true podCreationTimestamp="2026-09-16 00:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:52.997958078 +0000 UTC m=+291.541095054" watchObservedRunningTime="2026-09-16 00:47:52.998539214 +0000 UTC m=+291.541676190"
Sep 16 00:47:53.979795 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:53.979746    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:53.983206 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:53.983179    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"]
Sep 16 00:47:53.986525 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:53.986502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:53.989119 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:53.988722    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v2llq\""
Sep 16 00:47:53.989119 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:53.988780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 00:47:53.995475 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:53.994429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"]
Sep 16 00:47:54.089395 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.089357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c8d4a23-f845-47d0-b31f-6045a3154f7f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-v9t45\" (UID: \"3c8d4a23-f845-47d0-b31f-6045a3154f7f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:54.089587 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.089516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhg94\" (UniqueName: \"kubernetes.io/projected/3c8d4a23-f845-47d0-b31f-6045a3154f7f-kube-api-access-nhg94\") pod \"monitoring-plugin-5d85778b9d-v9t45\" (UID: \"3c8d4a23-f845-47d0-b31f-6045a3154f7f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:54.132558 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.132518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:54.190929 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.190888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c8d4a23-f845-47d0-b31f-6045a3154f7f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-v9t45\" (UID: \"3c8d4a23-f845-47d0-b31f-6045a3154f7f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:54.191116 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.191006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhg94\" (UniqueName: \"kubernetes.io/projected/3c8d4a23-f845-47d0-b31f-6045a3154f7f-kube-api-access-nhg94\") pod \"monitoring-plugin-5d85778b9d-v9t45\" (UID: \"3c8d4a23-f845-47d0-b31f-6045a3154f7f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:54.193373 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.193352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c8d4a23-f845-47d0-b31f-6045a3154f7f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-v9t45\" (UID: \"3c8d4a23-f845-47d0-b31f-6045a3154f7f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:54.199205 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.199180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhg94\" (UniqueName: \"kubernetes.io/projected/3c8d4a23-f845-47d0-b31f-6045a3154f7f-kube-api-access-nhg94\") pod \"monitoring-plugin-5d85778b9d-v9t45\" (UID: \"3c8d4a23-f845-47d0-b31f-6045a3154f7f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:54.300676 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.300639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:54.798377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.798353    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"]
Sep 16 00:47:54.800228 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:47:54.800202    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8d4a23_f845_47d0_b31f_6045a3154f7f.slice/crio-f06759ac107c8209ba68250148bb25119b75726943fbfbd05023233a28807cf1 WatchSource:0}: Error finding container f06759ac107c8209ba68250148bb25119b75726943fbfbd05023233a28807cf1: Status 404 returned error can't find the container with id f06759ac107c8209ba68250148bb25119b75726943fbfbd05023233a28807cf1
Sep 16 00:47:54.937049 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:47:54.937000    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mftnx" podUID="0d9cbade-9182-42ac-8e70-5fd8626eb8f3"
Sep 16 00:47:54.970974 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.970938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45" event={"ID":"3c8d4a23-f845-47d0-b31f-6045a3154f7f","Type":"ContainerStarted","Data":"f06759ac107c8209ba68250148bb25119b75726943fbfbd05023233a28807cf1"}
Sep 16 00:47:54.973814 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.973783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerStarted","Data":"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"}
Sep 16 00:47:54.973948 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.973840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerStarted","Data":"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"}
Sep 16 00:47:54.973948 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.973867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerStarted","Data":"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"}
Sep 16 00:47:54.973948 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.973892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerStarted","Data":"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"}
Sep 16 00:47:54.973948 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:54.973924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerStarted","Data":"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"}
Sep 16 00:47:55.406457 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:55.406384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:47:56.983093 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:56.983054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45" event={"ID":"3c8d4a23-f845-47d0-b31f-6045a3154f7f","Type":"ContainerStarted","Data":"18ebc2b336e5536735a87cc2b3f5a3fb1f37a12b6bed7028f867d57237eef1df"}
Sep 16 00:47:56.983616 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:56.983146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:56.987007 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:56.986982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerStarted","Data":"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"}
Sep 16 00:47:56.988197 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:56.988180    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45"
Sep 16 00:47:56.999887 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:56.999844    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-v9t45" podStartSLOduration=2.534470861 podStartE2EDuration="3.999831324s" totalImagesPullingTime="1.465360463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:56.999077242 +0000 UTC m=+295.542214219" watchObservedRunningTime="2026-09-16 00:47:56.999831324 +0000 UTC m=+295.542968353"
Sep 16 00:47:57.038208 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:47:57.038153    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.545131129 podStartE2EDuration="7.038134759s" totalImagesPullingTime="4.47853385s" totalInitContainerRuntime="14.46978ms" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:57.037713376 +0000 UTC m=+295.580850352" watchObservedRunningTime="2026-09-16 00:47:57.038134759 +0000 UTC m=+295.581271758"
Sep 16 00:48:00.955197 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:00.955167    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hs5c9"
Sep 16 00:48:01.836254 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:01.836222    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:48:01.838781 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:01.838754    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:48:01.847395 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:01.847370    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 00:48:01.941330 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:01.941208    2581 scope.go:117] "RemoveContainer" containerID="4d4d1fc189dcb12f0afe0cb1b169d31638185fd24e948ac538439835bcbbe573"
Sep 16 00:48:02.008107 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.008074    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:48:02.008524 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.008201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" event={"ID":"0187d13e-b0d9-40f1-91d1-0aea6ba2be15","Type":"ContainerStarted","Data":"82bf397865b27911912064231c9fad73db9135d2150291c01df0f6effdd285bb"}
Sep 16 00:48:02.008524 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.008513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:48:02.009713 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.009683    2581 patch_prober.go:28] interesting pod/console-operator-696884fb9c-c4jbm container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body=
Sep 16 00:48:02.009823 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.009732    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" podUID="0187d13e-b0d9-40f1-91d1-0aea6ba2be15" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused"
Sep 16 00:48:02.024601 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.024554    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm" podStartSLOduration=43.897363666 podStartE2EDuration="47.024535785s" totalImagesPullingTime="3.127172119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:02.023266641 +0000 UTC m=+300.566403631" watchObservedRunningTime="2026-09-16 00:48:02.024535785 +0000 UTC m=+300.567672759"
Sep 16 00:48:02.450761 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.450721    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:02.550451 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.550412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:02.612823 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:02.612790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:03.099013 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.098963    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:03.123728 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.123699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-c4jbm"
Sep 16 00:48:03.200431 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.200401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:03.293313 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.293267    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-889kn"]
Sep 16 00:48:03.296895 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.296870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-889kn"
Sep 16 00:48:03.299521 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.299475    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4hjfs\""
Sep 16 00:48:03.299774 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.299756    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 00:48:03.299960 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.299940    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 00:48:03.306533 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.306502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:03.307077 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.307048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-889kn"]
Sep 16 00:48:03.374336 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.374249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzlx\" (UniqueName: \"kubernetes.io/projected/7e70c35a-bd71-4fdf-adb8-8541124d237b-kube-api-access-xjzlx\") pod \"downloads-76f99584f4-889kn\" (UID: \"7e70c35a-bd71-4fdf-adb8-8541124d237b\") " pod="openshift-console/downloads-76f99584f4-889kn"
Sep 16 00:48:03.403397 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.403365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:03.475843 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.475798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzlx\" (UniqueName: \"kubernetes.io/projected/7e70c35a-bd71-4fdf-adb8-8541124d237b-kube-api-access-xjzlx\") pod \"downloads-76f99584f4-889kn\" (UID: \"7e70c35a-bd71-4fdf-adb8-8541124d237b\") " pod="openshift-console/downloads-76f99584f4-889kn"
Sep 16 00:48:03.484111 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.484081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzlx\" (UniqueName: \"kubernetes.io/projected/7e70c35a-bd71-4fdf-adb8-8541124d237b-kube-api-access-xjzlx\") pod \"downloads-76f99584f4-889kn\" (UID: \"7e70c35a-bd71-4fdf-adb8-8541124d237b\") " pod="openshift-console/downloads-76f99584f4-889kn"
Sep 16 00:48:03.502373 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.502339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:03.602271 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.602231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:03.608256 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.608224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-889kn"
Sep 16 00:48:03.891914 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.891888    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-889kn"]
Sep 16 00:48:03.893694 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:48:03.893667    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e70c35a_bd71_4fdf_adb8_8541124d237b.slice/crio-e5b0603d35ac394498d790dbe7bdf925ab954890c640c8cb015a248972fc674f WatchSource:0}: Error finding container e5b0603d35ac394498d790dbe7bdf925ab954890c640c8cb015a248972fc674f: Status 404 returned error can't find the container with id e5b0603d35ac394498d790dbe7bdf925ab954890c640c8cb015a248972fc674f
Sep 16 00:48:03.895630 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:03.895613    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:48:04.010293 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.010182    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:04.016167 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.016137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-889kn" event={"ID":"7e70c35a-bd71-4fdf-adb8-8541124d237b","Type":"ContainerStarted","Data":"e5b0603d35ac394498d790dbe7bdf925ab954890c640c8cb015a248972fc674f"}
Sep 16 00:48:04.110923 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.110890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:04.211282 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.211242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:04.261347 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.261252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:04.698792 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.698758    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78dfc67b84-z8qvb"]
Sep 16 00:48:04.703930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.703898    2581 patch_prober.go:28] interesting pod/image-registry-78dfc67b84-z8qvb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 00:48:04.704095 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.703968    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:48:04.784646 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.784614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:04.844002 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.843963    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:04.901747 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:04.901684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:05.185958 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:05.185923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:05.382159 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:05.382113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:05.654897 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:05.654858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:06.008754 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:06.008676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:06.108794 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:06.108747    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:08.935952 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:08.935639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:48:10.072050 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:10.072002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:10.122832 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:10.122800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:12.674387 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:12.674353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:14.180168 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.180135    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65fbbd55d9-j4rpq"]
Sep 16 00:48:14.184940 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.184913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.187356 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.187330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 00:48:14.188296 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.188271    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 00:48:14.188719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.188556    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 00:48:14.188719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.188632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m679f\""
Sep 16 00:48:14.188719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.188642    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 00:48:14.188719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.188679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 00:48:14.193528 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.193503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fbbd55d9-j4rpq"]
Sep 16 00:48:14.276547 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.276505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-oauth-config\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.276719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.276636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/4927ed83-6019-4e69-a127-1095f663a05e-kube-api-access-vjxdx\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.276719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.276660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-console-config\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.276719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.276679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-serving-cert\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.276719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.276708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-oauth-serving-cert\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.276900 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.276819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-service-ca\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.377751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.377707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-oauth-serving-cert\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.377945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.377771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-service-ca\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.377945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.377803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-oauth-config\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.377945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.377905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/4927ed83-6019-4e69-a127-1095f663a05e-kube-api-access-vjxdx\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.377945 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.377929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-console-config\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.378164 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.378053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-serving-cert\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.378604 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.378582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-service-ca\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.378775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.378754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-console-config\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.378837 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.378754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-oauth-serving-cert\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.380892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.380865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-oauth-config\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.381229 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.381203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-serving-cert\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.385884 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.385863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/4927ed83-6019-4e69-a127-1095f663a05e-kube-api-access-vjxdx\") pod \"console-65fbbd55d9-j4rpq\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") " pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.497198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.497099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:14.704603 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:14.704571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:48:17.592694 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:17.592653    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:17.650837 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:17.650801    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:18.605977 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:18.605933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:18.661524 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:18.661477    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:19.075881 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:19.075844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:19.134939 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:19.134907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:20.023685 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:20.023646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fbbd55d9-j4rpq"]
Sep 16 00:48:20.027810 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:48:20.027773    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4927ed83_6019_4e69_a127_1095f663a05e.slice/crio-9646c8c1001f642d4180baeb6a6479101635c47b6d2f2322856bfd9afd11f70d WatchSource:0}: Error finding container 9646c8c1001f642d4180baeb6a6479101635c47b6d2f2322856bfd9afd11f70d: Status 404 returned error can't find the container with id 9646c8c1001f642d4180baeb6a6479101635c47b6d2f2322856bfd9afd11f70d
Sep 16 00:48:20.087827 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:20.087777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-889kn" event={"ID":"7e70c35a-bd71-4fdf-adb8-8541124d237b","Type":"ContainerStarted","Data":"8f0e513df153d5058e2abbf88174b5c619e584425491ff0d0d34e49a47bf0385"}
Sep 16 00:48:20.088017 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:20.087991    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-889kn"
Sep 16 00:48:20.089324 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:20.089289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fbbd55d9-j4rpq" event={"ID":"4927ed83-6019-4e69-a127-1095f663a05e","Type":"ContainerStarted","Data":"9646c8c1001f642d4180baeb6a6479101635c47b6d2f2322856bfd9afd11f70d"}
Sep 16 00:48:20.097114 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:20.097084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-889kn"
Sep 16 00:48:20.104002 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:20.103952    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-889kn" podStartSLOduration=1.144706105 podStartE2EDuration="17.103935584s" totalImagesPullingTime="15.959229479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:20.103205097 +0000 UTC m=+318.646342072" watchObservedRunningTime="2026-09-16 00:48:20.103935584 +0000 UTC m=+318.647072602"
Sep 16 00:48:22.206400 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:22.206358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:22.303887 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:22.303846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:24.112816 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:24.112746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fbbd55d9-j4rpq" event={"ID":"4927ed83-6019-4e69-a127-1095f663a05e","Type":"ContainerStarted","Data":"a66fe3f74f7086c264ebf3b1ef76c4c77c7d6d7edea00778b1493cfd8fb22494"}
Sep 16 00:48:24.131917 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:24.131853    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65fbbd55d9-j4rpq" podStartSLOduration=6.26811762 podStartE2EDuration="10.131834631s" totalImagesPullingTime="3.863717011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:24.131234657 +0000 UTC m=+322.674371633" watchObservedRunningTime="2026-09-16 00:48:24.131834631 +0000 UTC m=+322.674971605"
Sep 16 00:48:24.497568 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:24.497524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:24.497751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:24.497603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:24.503091 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:24.503056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:25.122098 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:25.122070    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:25.608841 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:25.608797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:25.708775 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:25.708733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:26.609212 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:26.609170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:26.711511 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:26.711447    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:26.810414 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:26.810375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:27.605546 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:27.605502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:27.654631 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:27.654587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:29.411161 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:29.411127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:29.508452 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:29.508416    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:29.719607 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:29.719522    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a" containerName="registry" containerID="cri-o://440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204" gracePeriod=30
Sep 16 00:48:29.998407 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:29.998372    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:48:30.128856 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.128819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-installation-pull-secrets\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.128856 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.128862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-certificates\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.129112 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.128906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.129112 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.128932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-trusted-ca\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.129112 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.128948    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-image-registry-private-configuration\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.129112 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.128970    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-bound-sa-token\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.129112 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.129053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr9n2\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-kube-api-access-wr9n2\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.129112 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.129087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/339f2dfa-f2e9-4037-9bbd-84a53206887a-ca-trust-extracted\") pod \"339f2dfa-f2e9-4037-9bbd-84a53206887a\" (UID: \"339f2dfa-f2e9-4037-9bbd-84a53206887a\") "
Sep 16 00:48:30.129405 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.129367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:48:30.129405 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.129386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:48:30.131531 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.131455    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:48:30.131531 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.131516    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-kube-api-access-wr9n2" (OuterVolumeSpecName: "kube-api-access-wr9n2") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "kube-api-access-wr9n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:48:30.131868 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.131841    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:48:30.131953 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.131865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:48:30.131953 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.131938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:48:30.138858 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.138831    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/339f2dfa-f2e9-4037-9bbd-84a53206887a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "339f2dfa-f2e9-4037-9bbd-84a53206887a" (UID: "339f2dfa-f2e9-4037-9bbd-84a53206887a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:48:30.142264 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.142235    2581 generic.go:356] "Generic (PLEG): container finished" podID="339f2dfa-f2e9-4037-9bbd-84a53206887a" containerID="440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204" exitCode=0
Sep 16 00:48:30.142381 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.142278    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb"
Sep 16 00:48:30.142381 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.142353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" event={"ID":"339f2dfa-f2e9-4037-9bbd-84a53206887a","Type":"ContainerDied","Data":"440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204"}
Sep 16 00:48:30.142511 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.142400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dfc67b84-z8qvb" event={"ID":"339f2dfa-f2e9-4037-9bbd-84a53206887a","Type":"ContainerDied","Data":"8219f1e91a7e984b1ba9c8cb3e3ff8a0360995be0134f39caa1957b94ce8e221"}
Sep 16 00:48:30.142511 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.142428    2581 scope.go:117] "RemoveContainer" containerID="440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204"
Sep 16 00:48:30.151394 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.151375    2581 scope.go:117] "RemoveContainer" containerID="440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204"
Sep 16 00:48:30.151742 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:48:30.151718    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204\": container with ID starting with 440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204 not found: ID does not exist" containerID="440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204"
Sep 16 00:48:30.151842 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.151749    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204"} err="failed to get container status \"440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204\": rpc error: code = NotFound desc = could not find container \"440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204\": container with ID starting with 440c3d49c2c646dbeaad18ba6d279610bd803c597214e29b64cd32e8e6ac4204 not found: ID does not exist"
Sep 16 00:48:30.162168 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.162137    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78dfc67b84-z8qvb"]
Sep 16 00:48:30.165833 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.165809    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78dfc67b84-z8qvb"]
Sep 16 00:48:30.230284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230206    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-trusted-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:30.230284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230241    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-image-registry-private-configuration\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:30.230284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230251    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-bound-sa-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:30.230284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230261    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9n2\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-kube-api-access-wr9n2\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:30.230284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230271    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/339f2dfa-f2e9-4037-9bbd-84a53206887a-ca-trust-extracted\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:30.230284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230279    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/339f2dfa-f2e9-4037-9bbd-84a53206887a-installation-pull-secrets\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:30.230284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230288    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-certificates\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:30.230656 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:30.230297    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/339f2dfa-f2e9-4037-9bbd-84a53206887a-registry-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:31.940251 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:31.940214    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a" path="/var/lib/kubelet/pods/339f2dfa-f2e9-4037-9bbd-84a53206887a/volumes"
Sep 16 00:48:33.991144 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:33.991111    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fbbd55d9-j4rpq"]
Sep 16 00:48:34.128632 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:34.128594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:34.229764 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:34.229725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:34.278531 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:34.278500    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:36.167162 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:36.167125    2581 generic.go:356] "Generic (PLEG): container finished" podID="a63383b4-49c8-447e-982f-7cbf0fa0df04" containerID="4ff6845d8f61b95c4abc0a08de22575af483e00b808ab5025cea05bdbaaba081" exitCode=0
Sep 16 00:48:36.167563 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:36.167191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-q2972" event={"ID":"a63383b4-49c8-447e-982f-7cbf0fa0df04","Type":"ContainerDied","Data":"4ff6845d8f61b95c4abc0a08de22575af483e00b808ab5025cea05bdbaaba081"}
Sep 16 00:48:36.167613 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:36.167567    2581 scope.go:117] "RemoveContainer" containerID="4ff6845d8f61b95c4abc0a08de22575af483e00b808ab5025cea05bdbaaba081"
Sep 16 00:48:37.173245 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:37.173209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-q2972" event={"ID":"a63383b4-49c8-447e-982f-7cbf0fa0df04","Type":"ContainerStarted","Data":"44b1293a17ae022dd0bbad57ec97a43d2534644a935ff51aa1b99783894d8ca4"}
Sep 16 00:48:40.609298 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:40.609258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:40.668445 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:40.668389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:40.937727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:40.937627    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:40.986163 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:40.986107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:48:51.231935 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:51.231901    2581 generic.go:356] "Generic (PLEG): container finished" podID="485ae4b4-6055-4452-83ce-e1b618a2d23e" containerID="1045793e99d2aae8521ca8e3c91e18db14ac781fda2b2ce13254ac7dd1f3bf74" exitCode=0
Sep 16 00:48:51.232327 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:51.231969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml" event={"ID":"485ae4b4-6055-4452-83ce-e1b618a2d23e","Type":"ContainerDied","Data":"1045793e99d2aae8521ca8e3c91e18db14ac781fda2b2ce13254ac7dd1f3bf74"}
Sep 16 00:48:51.232327 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:51.232271    2581 scope.go:117] "RemoveContainer" containerID="1045793e99d2aae8521ca8e3c91e18db14ac781fda2b2ce13254ac7dd1f3bf74"
Sep 16 00:48:52.238144 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:52.238108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hwsml" event={"ID":"485ae4b4-6055-4452-83ce-e1b618a2d23e","Type":"ContainerStarted","Data":"f233b8112d31a8e957c80cce25bc137d18a72a4e7c9502eb651dd2e52616a9c4"}
Sep 16 00:48:59.017786 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.017736    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65fbbd55d9-j4rpq" podUID="4927ed83-6019-4e69-a127-1095f663a05e" containerName="console" containerID="cri-o://a66fe3f74f7086c264ebf3b1ef76c4c77c7d6d7edea00778b1493cfd8fb22494" gracePeriod=15
Sep 16 00:48:59.268457 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.268384    2581 generic.go:356] "Generic (PLEG): container finished" podID="4927ed83-6019-4e69-a127-1095f663a05e" containerID="a66fe3f74f7086c264ebf3b1ef76c4c77c7d6d7edea00778b1493cfd8fb22494" exitCode=0
Sep 16 00:48:59.268587 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.268460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fbbd55d9-j4rpq" event={"ID":"4927ed83-6019-4e69-a127-1095f663a05e","Type":"ContainerDied","Data":"a66fe3f74f7086c264ebf3b1ef76c4c77c7d6d7edea00778b1493cfd8fb22494"}
Sep 16 00:48:59.300156 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.300120    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:48:59.390600 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.390555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-console-config\") pod \"4927ed83-6019-4e69-a127-1095f663a05e\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") "
Sep 16 00:48:59.390600 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.390604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-oauth-serving-cert\") pod \"4927ed83-6019-4e69-a127-1095f663a05e\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") "
Sep 16 00:48:59.390846 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.390662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/4927ed83-6019-4e69-a127-1095f663a05e-kube-api-access-vjxdx\") pod \"4927ed83-6019-4e69-a127-1095f663a05e\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") "
Sep 16 00:48:59.390846 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.390677    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-service-ca\") pod \"4927ed83-6019-4e69-a127-1095f663a05e\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") "
Sep 16 00:48:59.390846 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.390717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-oauth-config\") pod \"4927ed83-6019-4e69-a127-1095f663a05e\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") "
Sep 16 00:48:59.390846 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.390752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-serving-cert\") pod \"4927ed83-6019-4e69-a127-1095f663a05e\" (UID: \"4927ed83-6019-4e69-a127-1095f663a05e\") "
Sep 16 00:48:59.391067 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.391039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-console-config" (OuterVolumeSpecName: "console-config") pod "4927ed83-6019-4e69-a127-1095f663a05e" (UID: "4927ed83-6019-4e69-a127-1095f663a05e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:48:59.391129 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.391049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4927ed83-6019-4e69-a127-1095f663a05e" (UID: "4927ed83-6019-4e69-a127-1095f663a05e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:48:59.391181 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.391134    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-service-ca" (OuterVolumeSpecName: "service-ca") pod "4927ed83-6019-4e69-a127-1095f663a05e" (UID: "4927ed83-6019-4e69-a127-1095f663a05e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:48:59.393006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.392983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4927ed83-6019-4e69-a127-1095f663a05e-kube-api-access-vjxdx" (OuterVolumeSpecName: "kube-api-access-vjxdx") pod "4927ed83-6019-4e69-a127-1095f663a05e" (UID: "4927ed83-6019-4e69-a127-1095f663a05e"). InnerVolumeSpecName "kube-api-access-vjxdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:48:59.393106 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.393040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4927ed83-6019-4e69-a127-1095f663a05e" (UID: "4927ed83-6019-4e69-a127-1095f663a05e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:48:59.393187 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.393114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4927ed83-6019-4e69-a127-1095f663a05e" (UID: "4927ed83-6019-4e69-a127-1095f663a05e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:48:59.491525 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.491462    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/4927ed83-6019-4e69-a127-1095f663a05e-kube-api-access-vjxdx\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:59.491525 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.491521    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:59.491525 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.491531    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-oauth-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:59.491749 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.491540    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4927ed83-6019-4e69-a127-1095f663a05e-console-serving-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:59.491749 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.491549    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-console-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:48:59.491749 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:48:59.491557    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4927ed83-6019-4e69-a127-1095f663a05e-oauth-serving-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:00.275860 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:00.275827    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fbbd55d9-j4rpq"
Sep 16 00:49:00.276288 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:00.275833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fbbd55d9-j4rpq" event={"ID":"4927ed83-6019-4e69-a127-1095f663a05e","Type":"ContainerDied","Data":"9646c8c1001f642d4180baeb6a6479101635c47b6d2f2322856bfd9afd11f70d"}
Sep 16 00:49:00.276288 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:00.275970    2581 scope.go:117] "RemoveContainer" containerID="a66fe3f74f7086c264ebf3b1ef76c4c77c7d6d7edea00778b1493cfd8fb22494"
Sep 16 00:49:00.292228 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:00.292198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fbbd55d9-j4rpq"]
Sep 16 00:49:00.294141 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:00.294119    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65fbbd55d9-j4rpq"]
Sep 16 00:49:01.939451 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:01.939417    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4927ed83-6019-4e69-a127-1095f663a05e" path="/var/lib/kubelet/pods/4927ed83-6019-4e69-a127-1095f663a05e/volumes"
Sep 16 00:49:08.429615 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:08.429573    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:09.135813 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.135781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:09.494204 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.494112    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:49:09.494703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.494462    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="alertmanager" containerID="cri-o://b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602" gracePeriod=120
Sep 16 00:49:09.494703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.494511    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-metric" containerID="cri-o://f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942" gracePeriod=120
Sep 16 00:49:09.494703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.494531    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-web" containerID="cri-o://b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679" gracePeriod=120
Sep 16 00:49:09.494703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.494559    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy" containerID="cri-o://c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88" gracePeriod=120
Sep 16 00:49:09.494703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.494591    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="prom-label-proxy" containerID="cri-o://9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2" gracePeriod=120
Sep 16 00:49:09.494703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:09.494590    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="config-reloader" containerID="cri-o://f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15" gracePeriod=120
Sep 16 00:49:10.317141 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317106    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerID="9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2" exitCode=0
Sep 16 00:49:10.317141 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317133    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerID="c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88" exitCode=0
Sep 16 00:49:10.317141 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317140    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerID="f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15" exitCode=0
Sep 16 00:49:10.317141 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317146    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerID="b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602" exitCode=0
Sep 16 00:49:10.317407 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"}
Sep 16 00:49:10.317407 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"}
Sep 16 00:49:10.317407 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"}
Sep 16 00:49:10.317407 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.317288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"}
Sep 16 00:49:10.729184 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.729160    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:10.893805 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.893694    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-volume\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.893805 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.893769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m5sf\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-kube-api-access-6m5sf\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.893805 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.893795    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-out\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.893805 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.893812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-main-tls\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.893830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-tls-assets\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.893854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-main-db\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.893877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-metrics-client-ca\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-metric\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894104    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-cluster-tls-config\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894411 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-web\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894411 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894411 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894248    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-web-config\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894411 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894280    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-trusted-ca-bundle\") pod \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\" (UID: \"eba251c2-2caa-42d6-91ed-ac7eaa417a92\") "
Sep 16 00:49:10.894411 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894368    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:49:10.894690 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894656    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-main-db\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.894810 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:10.894965 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.894924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:10.896809 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.896770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-kube-api-access-6m5sf" (OuterVolumeSpecName: "kube-api-access-6m5sf") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "kube-api-access-6m5sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:49:10.896927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.896836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-out" (OuterVolumeSpecName: "config-out") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:49:10.897511 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.897452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-volume" (OuterVolumeSpecName: "config-volume") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:10.897620 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.897555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:10.897620 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.897568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:10.898307 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.898172    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:10.898307 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.898259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:49:10.898307 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.898278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:10.901819 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.901762    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:10.908102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.908080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-web-config" (OuterVolumeSpecName: "web-config") pod "eba251c2-2caa-42d6-91ed-ac7eaa417a92" (UID: "eba251c2-2caa-42d6-91ed-ac7eaa417a92"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:10.996040 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.995990    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996040 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996034    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-cluster-tls-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996040 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996045    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996040 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996054    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996062    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-web-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996072    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996080    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-volume\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996088    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m5sf\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-kube-api-access-6m5sf\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996096    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba251c2-2caa-42d6-91ed-ac7eaa417a92-config-out\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996105    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eba251c2-2caa-42d6-91ed-ac7eaa417a92-secret-alertmanager-main-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996112    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba251c2-2caa-42d6-91ed-ac7eaa417a92-tls-assets\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:10.996308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:10.996120    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba251c2-2caa-42d6-91ed-ac7eaa417a92-metrics-client-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:49:11.324654 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.324620    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerID="f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942" exitCode=0
Sep 16 00:49:11.324654 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.324645    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerID="b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679" exitCode=0
Sep 16 00:49:11.324907 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.324704    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.324907 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.324710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"}
Sep 16 00:49:11.324907 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.324795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"}
Sep 16 00:49:11.324907 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.324835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eba251c2-2caa-42d6-91ed-ac7eaa417a92","Type":"ContainerDied","Data":"0cfd80fa100a7924f731da39c12e9fb5e5d1e57caa88518491d05d3bd871a08b"}
Sep 16 00:49:11.324907 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.324858    2581 scope.go:117] "RemoveContainer" containerID="9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"
Sep 16 00:49:11.333028 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.333010    2581 scope.go:117] "RemoveContainer" containerID="f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"
Sep 16 00:49:11.339972 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.339950    2581 scope.go:117] "RemoveContainer" containerID="c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"
Sep 16 00:49:11.347085 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.347069    2581 scope.go:117] "RemoveContainer" containerID="b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"
Sep 16 00:49:11.347584 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.347552    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:49:11.351412 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.351377    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:49:11.354554 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.354535    2581 scope.go:117] "RemoveContainer" containerID="f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"
Sep 16 00:49:11.361230 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.361214    2581 scope.go:117] "RemoveContainer" containerID="b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"
Sep 16 00:49:11.368790 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.368772    2581 scope.go:117] "RemoveContainer" containerID="cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7"
Sep 16 00:49:11.375086 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375064    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:49:11.375367 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375354    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="init-config-reloader"
Sep 16 00:49:11.375408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375368    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="init-config-reloader"
Sep 16 00:49:11.375408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375378    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4927ed83-6019-4e69-a127-1095f663a05e" containerName="console"
Sep 16 00:49:11.375408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4927ed83-6019-4e69-a127-1095f663a05e" containerName="console"
Sep 16 00:49:11.375408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375390    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a" containerName="registry"
Sep 16 00:49:11.375408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375395    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a" containerName="registry"
Sep 16 00:49:11.375408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-metric"
Sep 16 00:49:11.375408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375409    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-metric"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375415    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="config-reloader"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375420    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="config-reloader"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375426    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="alertmanager"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375431    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="alertmanager"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375437    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-web"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375442    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-web"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375447    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375452    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375457    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="prom-label-proxy"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375462    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="prom-label-proxy"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375530    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4927ed83-6019-4e69-a127-1095f663a05e" containerName="console"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375539    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375545    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="config-reloader"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375552    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="prom-label-proxy"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375559    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="339f2dfa-f2e9-4037-9bbd-84a53206887a" containerName="registry"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375564    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="alertmanager"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375570    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-metric"
Sep 16 00:49:11.375675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.375577    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" containerName="kube-rbac-proxy-web"
Sep 16 00:49:11.377725 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.377495    2581 scope.go:117] "RemoveContainer" containerID="9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"
Sep 16 00:49:11.378640 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:11.378612    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2\": container with ID starting with 9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2 not found: ID does not exist" containerID="9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"
Sep 16 00:49:11.378729 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.378654    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"} err="failed to get container status \"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2\": rpc error: code = NotFound desc = could not find container \"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2\": container with ID starting with 9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2 not found: ID does not exist"
Sep 16 00:49:11.378729 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.378679    2581 scope.go:117] "RemoveContainer" containerID="f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:11.379328    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942\": container with ID starting with f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942 not found: ID does not exist" containerID="f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.379360    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"} err="failed to get container status \"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942\": rpc error: code = NotFound desc = could not find container \"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942\": container with ID starting with f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942 not found: ID does not exist"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.379382    2581 scope.go:117] "RemoveContainer" containerID="c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:11.379644    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88\": container with ID starting with c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88 not found: ID does not exist" containerID="c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.379667    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"} err="failed to get container status \"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88\": rpc error: code = NotFound desc = could not find container \"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88\": container with ID starting with c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88 not found: ID does not exist"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.379685    2581 scope.go:117] "RemoveContainer" containerID="b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:11.379897    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679\": container with ID starting with b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679 not found: ID does not exist" containerID="b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.379920    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"} err="failed to get container status \"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679\": rpc error: code = NotFound desc = could not find container \"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679\": container with ID starting with b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679 not found: ID does not exist"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.379936    2581 scope.go:117] "RemoveContainer" containerID="f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:11.380141    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15\": container with ID starting with f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15 not found: ID does not exist" containerID="f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380162    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"} err="failed to get container status \"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15\": rpc error: code = NotFound desc = could not find container \"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15\": container with ID starting with f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15 not found: ID does not exist"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380180    2581 scope.go:117] "RemoveContainer" containerID="b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:11.380377    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602\": container with ID starting with b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602 not found: ID does not exist" containerID="b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380398    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"} err="failed to get container status \"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602\": rpc error: code = NotFound desc = could not find container \"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602\": container with ID starting with b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602 not found: ID does not exist"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380415    2581 scope.go:117] "RemoveContainer" containerID="cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7"
Sep 16 00:49:11.380624 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:11.380637    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7\": container with ID starting with cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7 not found: ID does not exist" containerID="cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380661    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7"} err="failed to get container status \"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7\": rpc error: code = NotFound desc = could not find container \"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7\": container with ID starting with cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7 not found: ID does not exist"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380678    2581 scope.go:117] "RemoveContainer" containerID="9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380953    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2"} err="failed to get container status \"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2\": rpc error: code = NotFound desc = could not find container \"9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2\": container with ID starting with 9cd0c71053b544c41c1d21ffa9b291079ab0262faa58c2b2987b80e6ac84f1d2 not found: ID does not exist"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.380974    2581 scope.go:117] "RemoveContainer" containerID="f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.381296    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942"} err="failed to get container status \"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942\": rpc error: code = NotFound desc = could not find container \"f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942\": container with ID starting with f3bd364b34422e42c92011944a1d8db8a1a3240f80244946003fe31dcf46f942 not found: ID does not exist"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.381318    2581 scope.go:117] "RemoveContainer" containerID="c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.381680    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88"} err="failed to get container status \"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88\": rpc error: code = NotFound desc = could not find container \"c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88\": container with ID starting with c7732ef2632a84203d8b1b4cc3a2023277fd5109ed40268af47e30d7fc9cdd88 not found: ID does not exist"
Sep 16 00:49:11.381769 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.381708    2581 scope.go:117] "RemoveContainer" containerID="b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"
Sep 16 00:49:11.382101 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.381785    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.382101 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.381996    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679"} err="failed to get container status \"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679\": rpc error: code = NotFound desc = could not find container \"b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679\": container with ID starting with b4445d6184d4dd47bba046828c42d1408d094224b453ebb581211c188fe1c679 not found: ID does not exist"
Sep 16 00:49:11.382101 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.382019    2581 scope.go:117] "RemoveContainer" containerID="f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"
Sep 16 00:49:11.382398 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.382376    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15"} err="failed to get container status \"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15\": rpc error: code = NotFound desc = could not find container \"f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15\": container with ID starting with f4bae15bbbfd87f0b545fe40a9c424496479fcb5b3565cbdeeccc28e539bcd15 not found: ID does not exist"
Sep 16 00:49:11.382456 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.382399    2581 scope.go:117] "RemoveContainer" containerID="b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"
Sep 16 00:49:11.382684 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.382660    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602"} err="failed to get container status \"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602\": rpc error: code = NotFound desc = could not find container \"b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602\": container with ID starting with b99e6ca787f5466050de180b51f9683bf9d14fde6e1855a1b405e444c17e3602 not found: ID does not exist"
Sep 16 00:49:11.382737 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.382687    2581 scope.go:117] "RemoveContainer" containerID="cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7"
Sep 16 00:49:11.382898 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.382878    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7"} err="failed to get container status \"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7\": rpc error: code = NotFound desc = could not find container \"cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7\": container with ID starting with cd4d2e5105e098466c26d5a19f282eb6a5a3b84ca4e4fb70f83afd8e68a067a7 not found: ID does not exist"
Sep 16 00:49:11.383873 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.383853    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 00:49:11.384074 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384042    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 00:49:11.384181 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384110    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 00:49:11.384181 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 00:49:11.384290 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384188    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 00:49:11.384333 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 00:49:11.384555 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 00:49:11.384690 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384572    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 00:49:11.384690 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.384624    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6qrrx\""
Sep 16 00:49:11.389661 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.389642    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 00:49:11.394620 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.394599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:49:11.501764 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.501930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqqf\" (UniqueName: \"kubernetes.io/projected/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-kube-api-access-mqqqf\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.501930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.501930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-config-volume\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.501930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502069 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502069 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502069 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.501985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502069 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.502008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-web-config\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502069 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.502030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502069 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.502046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502250 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.502069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-config-out\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.502250 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.502120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.602716 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.602875 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.602875 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-web-config\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.602875 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.602875 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.602875 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-config-out\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.603220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.603220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.602932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.603220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.603043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqqf\" (UniqueName: \"kubernetes.io/projected/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-kube-api-access-mqqqf\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.603220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.603073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.603220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.603117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-config-volume\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.603220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.603172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.603220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.603201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.604151 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.603925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.604151 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.603985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.604573 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.604167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.605904 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.605875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.606003 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.605987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.606079 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.606017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-config-out\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.606332 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.606293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.606664 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.606643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-config-volume\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.606765 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.606745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.606961 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.606943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-web-config\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.607207 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.607186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.607507 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.607468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.611336 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.611318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqqf\" (UniqueName: \"kubernetes.io/projected/7229b5f4-c8ec-409b-9a7a-3b34b04c1a15-kube-api-access-mqqqf\") pod \"alertmanager-main-0\" (UID: \"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.692031 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.691990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 00:49:11.940310 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.940222    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba251c2-2caa-42d6-91ed-ac7eaa417a92" path="/var/lib/kubelet/pods/eba251c2-2caa-42d6-91ed-ac7eaa417a92/volumes"
Sep 16 00:49:11.969986 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:11.969892    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 00:49:11.972230 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:49:11.972202    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7229b5f4_c8ec_409b_9a7a_3b34b04c1a15.slice/crio-28d3662f36792e0f71d6eed4fc8ac1649f44b1db93b2d9c7e6b37c414c423698 WatchSource:0}: Error finding container 28d3662f36792e0f71d6eed4fc8ac1649f44b1db93b2d9c7e6b37c414c423698: Status 404 returned error can't find the container with id 28d3662f36792e0f71d6eed4fc8ac1649f44b1db93b2d9c7e6b37c414c423698
Sep 16 00:49:12.332087 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:12.332050    2581 generic.go:356] "Generic (PLEG): container finished" podID="7229b5f4-c8ec-409b-9a7a-3b34b04c1a15" containerID="181496c738278e754c2d3c4b718593444ebdd54d5758bbf195457199bb748003" exitCode=0
Sep 16 00:49:12.332293 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:12.332120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerDied","Data":"181496c738278e754c2d3c4b718593444ebdd54d5758bbf195457199bb748003"}
Sep 16 00:49:12.332293 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:12.332169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerStarted","Data":"28d3662f36792e0f71d6eed4fc8ac1649f44b1db93b2d9c7e6b37c414c423698"}
Sep 16 00:49:12.664009 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:49:12.663982    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7229b5f4_c8ec_409b_9a7a_3b34b04c1a15.slice/crio-53b0e5dfa3ba3aab790840156b19d5dee794fe059f6f8f2831a72e1997562ac5.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 00:49:13.338924 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.338889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerStarted","Data":"53b0e5dfa3ba3aab790840156b19d5dee794fe059f6f8f2831a72e1997562ac5"}
Sep 16 00:49:13.339310 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.338982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerStarted","Data":"98a27d5f20dfb085cfaf3b13f6ab795fcdaeadefb04d513fc5c0828a82ae73ac"}
Sep 16 00:49:13.339310 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.339016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerStarted","Data":"8d3dd3035e7e3d0dbba68d96232d18f9b64ce9302021d62ebe3bd21bc8d4485b"}
Sep 16 00:49:13.339310 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.339043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerStarted","Data":"85f8883c8057cf4ea1d48b5a10a07bd9b2f9b180276b7964de45fd65cab9d590"}
Sep 16 00:49:13.339310 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.339074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerStarted","Data":"0f7857be19d4462bae98a409bacb79404bdf7a93fbf23150ebaecd99b3df6c37"}
Sep 16 00:49:13.339310 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.339096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7229b5f4-c8ec-409b-9a7a-3b34b04c1a15","Type":"ContainerStarted","Data":"e861a98679d29ba26f12e6ac34ed201f636aec1b6770699e82e448c531815c5b"}
Sep 16 00:49:13.368663 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.368611    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.349305726 podStartE2EDuration="2.368592734s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.287008ms" isStatefulPod=false podCreationTimestamp="2026-09-16 00:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:49:13.366503383 +0000 UTC m=+371.909640360" watchObservedRunningTime="2026-09-16 00:49:13.368592734 +0000 UTC m=+371.911729709"
Sep 16 00:49:13.551794 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.551757    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-587678f467-2xt4g"]
Sep 16 00:49:13.554779 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.554751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.557156 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.557131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4tkt7\""
Sep 16 00:49:13.557375 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.557241    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 00:49:13.557670 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.557267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 00:49:13.557934 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.557265    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 00:49:13.558308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.557347    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 00:49:13.558308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.557362    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 00:49:13.562999 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.562907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 00:49:13.567714 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.567685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-587678f467-2xt4g"]
Sep 16 00:49:13.621093 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-telemeter-client-tls\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.621093 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-federate-client-tls\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.621274 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxp9p\" (UniqueName: \"kubernetes.io/projected/2b9bfcc2-9fb6-4719-af15-270818df9acf-kube-api-access-lxp9p\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.621274 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-secret-telemeter-client\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.621274 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.621274 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-metrics-client-ca\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.621274 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.621274 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.621262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-serving-certs-ca-bundle\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.721911 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.721862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.721935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-metrics-client-ca\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.721963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.721997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-serving-certs-ca-bundle\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.722027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-telemeter-client-tls\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.722064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-federate-client-tls\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722524 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.722116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxp9p\" (UniqueName: \"kubernetes.io/projected/2b9bfcc2-9fb6-4719-af15-270818df9acf-kube-api-access-lxp9p\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722671 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.722565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-secret-telemeter-client\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.722964 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.722939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-serving-certs-ca-bundle\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.723040 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.723025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.723455 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.723428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bfcc2-9fb6-4719-af15-270818df9acf-metrics-client-ca\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.724834 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.724809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.724986 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.724959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-telemeter-client-tls\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.725106 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.725089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-federate-client-tls\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.725650 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.725631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2b9bfcc2-9fb6-4719-af15-270818df9acf-secret-telemeter-client\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.731261 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.731233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxp9p\" (UniqueName: \"kubernetes.io/projected/2b9bfcc2-9fb6-4719-af15-270818df9acf-kube-api-access-lxp9p\") pod \"telemeter-client-587678f467-2xt4g\" (UID: \"2b9bfcc2-9fb6-4719-af15-270818df9acf\") " pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:13.866333 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:13.866288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-587678f467-2xt4g"
Sep 16 00:49:14.158436 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:14.158402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-587678f467-2xt4g"]
Sep 16 00:49:14.159401 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:49:14.159364    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9bfcc2_9fb6_4719_af15_270818df9acf.slice/crio-d3ac031950fc1ca2edc285c12771ef755a2a39597e4363854da7fa905b4809f5 WatchSource:0}: Error finding container d3ac031950fc1ca2edc285c12771ef755a2a39597e4363854da7fa905b4809f5: Status 404 returned error can't find the container with id d3ac031950fc1ca2edc285c12771ef755a2a39597e4363854da7fa905b4809f5
Sep 16 00:49:14.343475 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:14.343437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587678f467-2xt4g" event={"ID":"2b9bfcc2-9fb6-4719-af15-270818df9acf","Type":"ContainerStarted","Data":"d3ac031950fc1ca2edc285c12771ef755a2a39597e4363854da7fa905b4809f5"}
Sep 16 00:49:14.832585 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:14.832541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:49:14.835321 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:14.835289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cbade-9182-42ac-8e70-5fd8626eb8f3-metrics-certs\") pod \"network-metrics-daemon-mftnx\" (UID: \"0d9cbade-9182-42ac-8e70-5fd8626eb8f3\") " pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:49:14.939568 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:14.939532    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6btht\""
Sep 16 00:49:14.947404 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:14.947381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mftnx"
Sep 16 00:49:15.224150 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:15.224067    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mftnx"]
Sep 16 00:49:15.227173 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:49:15.227140    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9cbade_9182_42ac_8e70_5fd8626eb8f3.slice/crio-911c6359cb4912dabc78ed1c539305a3df5a951e5fee3a5750f4c32066155cee WatchSource:0}: Error finding container 911c6359cb4912dabc78ed1c539305a3df5a951e5fee3a5750f4c32066155cee: Status 404 returned error can't find the container with id 911c6359cb4912dabc78ed1c539305a3df5a951e5fee3a5750f4c32066155cee
Sep 16 00:49:15.351722 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:15.351670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mftnx" event={"ID":"0d9cbade-9182-42ac-8e70-5fd8626eb8f3","Type":"ContainerStarted","Data":"911c6359cb4912dabc78ed1c539305a3df5a951e5fee3a5750f4c32066155cee"}
Sep 16 00:49:16.357736 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.357701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587678f467-2xt4g" event={"ID":"2b9bfcc2-9fb6-4719-af15-270818df9acf","Type":"ContainerStarted","Data":"a4a7432f641c10143620eaf66fe004d5c5374032b140e03fad5e96b52205d7c0"}
Sep 16 00:49:16.358194 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.357764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587678f467-2xt4g" event={"ID":"2b9bfcc2-9fb6-4719-af15-270818df9acf","Type":"ContainerStarted","Data":"9ba0e26bd64f9e8ce71a89805901b276f4f1ba9b83faa1a14cb6b22931f98c35"}
Sep 16 00:49:16.358194 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.357789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587678f467-2xt4g" event={"ID":"2b9bfcc2-9fb6-4719-af15-270818df9acf","Type":"ContainerStarted","Data":"a0a57c57acb2324734c35d9a7ab900e262b2a050c3aa60d1ba9460fb39416d91"}
Sep 16 00:49:16.380984 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.380911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-587678f467-2xt4g" podStartSLOduration=1.6744807000000002 podStartE2EDuration="3.380886958s" totalImagesPullingTime="1.706406258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:49:16.378227007 +0000 UTC m=+374.921364017" watchObservedRunningTime="2026-09-16 00:49:16.380886958 +0000 UTC m=+374.924023935"
Sep 16 00:49:16.462545 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.462506    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d66f48fc8-zg4m9"]
Sep 16 00:49:16.465780 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.465754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.468035 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.468008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 00:49:16.468035 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.468014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m679f\""
Sep 16 00:49:16.468219 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.468017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 00:49:16.468219 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.468102    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 00:49:16.468219 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.468017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 00:49:16.468421 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.468400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 00:49:16.474920 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.474726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 00:49:16.475961 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.475936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d66f48fc8-zg4m9"]
Sep 16 00:49:16.512747 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.512702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:16.546431 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.546391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-trusted-ca-bundle\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.546431 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.546437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t657z\" (UniqueName: \"kubernetes.io/projected/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-kube-api-access-t657z\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.546696 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.546546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-oauth-config\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.546696 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.546591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-serving-cert\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.546696 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.546608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-config\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.546696 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.546633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-oauth-serving-cert\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.546904 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.546707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-service-ca\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.647857 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.647831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-serving-cert\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.647971 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.647871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-config\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.647971 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.647954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-oauth-serving-cert\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.648089 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.648003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-service-ca\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.648089 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.648062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-trusted-ca-bundle\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.648187 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.648086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t657z\" (UniqueName: \"kubernetes.io/projected/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-kube-api-access-t657z\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.648187 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.648162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-oauth-config\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.650255 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.649193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-service-ca\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.650255 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.649719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-oauth-serving-cert\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.650429 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.650397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-trusted-ca-bundle\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.650889 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.650868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-serving-cert\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.651136 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.651103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-config\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.652978 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.652956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-oauth-config\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.654542 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.654469    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:16.657462 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.657440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t657z\" (UniqueName: \"kubernetes.io/projected/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-kube-api-access-t657z\") pod \"console-d66f48fc8-zg4m9\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") " pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:16.780386 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:16.780341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:17.072987 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.072629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d66f48fc8-zg4m9"]
Sep 16 00:49:17.075793 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:49:17.075760    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e88b26_ca27_45cf_83c2_5a84bb0c7e82.slice/crio-131a475c5fe72030d77f09104d7414576cb2bd5f5ac0876de41e4c186eae6b26 WatchSource:0}: Error finding container 131a475c5fe72030d77f09104d7414576cb2bd5f5ac0876de41e4c186eae6b26: Status 404 returned error can't find the container with id 131a475c5fe72030d77f09104d7414576cb2bd5f5ac0876de41e4c186eae6b26
Sep 16 00:49:17.253821 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.253789    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:17.303184 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.303157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:17.364229 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.364148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66f48fc8-zg4m9" event={"ID":"87e88b26-ca27-45cf-83c2-5a84bb0c7e82","Type":"ContainerStarted","Data":"959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84"}
Sep 16 00:49:17.364229 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.364203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66f48fc8-zg4m9" event={"ID":"87e88b26-ca27-45cf-83c2-5a84bb0c7e82","Type":"ContainerStarted","Data":"131a475c5fe72030d77f09104d7414576cb2bd5f5ac0876de41e4c186eae6b26"}
Sep 16 00:49:17.366144 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.366118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mftnx" event={"ID":"0d9cbade-9182-42ac-8e70-5fd8626eb8f3","Type":"ContainerStarted","Data":"78174ed7d686b1c052f61ed79943de8615bb0cc62eb37da81815233adca04e17"}
Sep 16 00:49:17.366261 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.366169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mftnx" event={"ID":"0d9cbade-9182-42ac-8e70-5fd8626eb8f3","Type":"ContainerStarted","Data":"af8b1cf005e425653e2d6ab62b072e0f8f317116f2b265da5e9195e5cda9c7ba"}
Sep 16 00:49:17.379710 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.379666    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d66f48fc8-zg4m9" podStartSLOduration=1.379654116 podStartE2EDuration="1.379654116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:49:17.378809629 +0000 UTC m=+375.921946631" watchObservedRunningTime="2026-09-16 00:49:17.379654116 +0000 UTC m=+375.922791091"
Sep 16 00:49:17.393344 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.393306    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mftnx" podStartSLOduration=373.97463165 podStartE2EDuration="6m15.393291331s" totalImagesPullingTime="1.418659681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:49:17.392527744 +0000 UTC m=+375.935664720" watchObservedRunningTime="2026-09-16 00:49:17.393291331 +0000 UTC m=+375.936428306"
Sep 16 00:49:17.472951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.472918    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:17.528876 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:17.528849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:22.204897 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:22.204857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:22.262426 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:22.262396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:26.781545 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:26.781509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:26.781925 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:26.781565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:26.786235 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:26.786210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:27.411285 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:27.411256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:49:27.548570 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:27.548535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:27.650287 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:27.650250    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:49:27.702198 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:49:27.702114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:01.830405 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:01.830368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:04.496437 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:04.496401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:06.913568 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:06.913522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:23.651833 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:23.651792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:23.705969 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:23.705929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:25.970968 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:25.970934    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-567bdd56d6-727qs"]
Sep 16 00:50:25.973502 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:25.973458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:25.985000 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:25.984976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567bdd56d6-727qs"]
Sep 16 00:50:26.023434 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.023394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:26.031959 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.031931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-service-ca\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.032122 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.031966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrqs\" (UniqueName: \"kubernetes.io/projected/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-kube-api-access-wzrqs\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.032122 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.031990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-oauth-config\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.032122 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.032004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-oauth-serving-cert\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.032122 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.032022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-config\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.032305 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.032134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-serving-cert\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.032305 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.032197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-trusted-ca-bundle\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.073595 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.073560    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:26.133392 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.133353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-oauth-config\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.133392 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.133398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-oauth-serving-cert\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.133681 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.133427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-config\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.133681 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.133460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-serving-cert\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.133681 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.133522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-trusted-ca-bundle\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.133814 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.133723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-service-ca\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.133814 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.133778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrqs\" (UniqueName: \"kubernetes.io/projected/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-kube-api-access-wzrqs\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.134295 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.134233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-oauth-serving-cert\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.134295 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.134283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-config\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.134449 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.134370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-trusted-ca-bundle\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.134449 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.134431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-service-ca\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.136006 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.135988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-serving-cert\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.136123 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.136048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-oauth-config\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.143308 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.143286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrqs\" (UniqueName: \"kubernetes.io/projected/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-kube-api-access-wzrqs\") pod \"console-567bdd56d6-727qs\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") " pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.167021 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.166993    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:26.218715 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.218675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:26.283767 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.283739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:26.552472 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.552383    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567bdd56d6-727qs"]
Sep 16 00:50:26.555523 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:50:26.555494    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3b6fd3_450f_4b2e_bc08_219245bcc83c.slice/crio-d76e4d831554e8fbf53fc373f2e720681e188a5982fe7d02dce1e52343dd1b92 WatchSource:0}: Error finding container d76e4d831554e8fbf53fc373f2e720681e188a5982fe7d02dce1e52343dd1b92: Status 404 returned error can't find the container with id d76e4d831554e8fbf53fc373f2e720681e188a5982fe7d02dce1e52343dd1b92
Sep 16 00:50:26.642145 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.642103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bdd56d6-727qs" event={"ID":"4b3b6fd3-450f-4b2e-bc08-219245bcc83c","Type":"ContainerStarted","Data":"76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51"}
Sep 16 00:50:26.642357 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.642168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bdd56d6-727qs" event={"ID":"4b3b6fd3-450f-4b2e-bc08-219245bcc83c","Type":"ContainerStarted","Data":"d76e4d831554e8fbf53fc373f2e720681e188a5982fe7d02dce1e52343dd1b92"}
Sep 16 00:50:26.659339 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.659290    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-567bdd56d6-727qs" podStartSLOduration=1.659274013 podStartE2EDuration="1.659274013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:50:26.658509383 +0000 UTC m=+445.201646359" watchObservedRunningTime="2026-09-16 00:50:26.659274013 +0000 UTC m=+445.202410987"
Sep 16 00:50:26.773655 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:26.773618    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:32.705577 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:32.705533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:32.757171 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:32.757144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:35.615301 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:35.615249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:35.668084 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:35.668044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:36.284302 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:36.284269    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:36.284503 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:36.284325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:36.289090 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:36.289068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:36.684580 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:36.684473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:50:36.727288 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:36.727246    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d66f48fc8-zg4m9"]
Sep 16 00:50:36.853784 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:36.853751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:36.955075 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:36.954974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:37.032633 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:37.032599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:40.611343 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:40.611305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:50:40.711322 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:50:40.711268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:01.752771 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:01.752697    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d66f48fc8-zg4m9" podUID="87e88b26-ca27-45cf-83c2-5a84bb0c7e82" containerName="console" containerID="cri-o://959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84" gracePeriod=15
Sep 16 00:51:01.992760 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:01.992733    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:51:02.143274 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-serving-cert\") pod \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") "
Sep 16 00:51:02.143473 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143292    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-oauth-serving-cert\") pod \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") "
Sep 16 00:51:02.143473 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143317    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t657z\" (UniqueName: \"kubernetes.io/projected/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-kube-api-access-t657z\") pod \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") "
Sep 16 00:51:02.143473 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-config\") pod \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") "
Sep 16 00:51:02.143473 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143360    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-trusted-ca-bundle\") pod \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") "
Sep 16 00:51:02.143473 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-service-ca\") pod \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") "
Sep 16 00:51:02.143473 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-oauth-config\") pod \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\" (UID: \"87e88b26-ca27-45cf-83c2-5a84bb0c7e82\") "
Sep 16 00:51:02.143908 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "87e88b26-ca27-45cf-83c2-5a84bb0c7e82" (UID: "87e88b26-ca27-45cf-83c2-5a84bb0c7e82"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:51:02.143908 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143895    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-config" (OuterVolumeSpecName: "console-config") pod "87e88b26-ca27-45cf-83c2-5a84bb0c7e82" (UID: "87e88b26-ca27-45cf-83c2-5a84bb0c7e82"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:51:02.144051 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "87e88b26-ca27-45cf-83c2-5a84bb0c7e82" (UID: "87e88b26-ca27-45cf-83c2-5a84bb0c7e82"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:51:02.144051 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.143916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-service-ca" (OuterVolumeSpecName: "service-ca") pod "87e88b26-ca27-45cf-83c2-5a84bb0c7e82" (UID: "87e88b26-ca27-45cf-83c2-5a84bb0c7e82"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:51:02.144155 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.144082    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:02.144155 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.144097    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-oauth-serving-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:02.144155 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.144106    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:02.144155 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.144116    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-trusted-ca-bundle\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:02.145591 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.145564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-kube-api-access-t657z" (OuterVolumeSpecName: "kube-api-access-t657z") pod "87e88b26-ca27-45cf-83c2-5a84bb0c7e82" (UID: "87e88b26-ca27-45cf-83c2-5a84bb0c7e82"). InnerVolumeSpecName "kube-api-access-t657z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:51:02.145591 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.145587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "87e88b26-ca27-45cf-83c2-5a84bb0c7e82" (UID: "87e88b26-ca27-45cf-83c2-5a84bb0c7e82"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:51:02.145722 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.145610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "87e88b26-ca27-45cf-83c2-5a84bb0c7e82" (UID: "87e88b26-ca27-45cf-83c2-5a84bb0c7e82"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:51:02.245125 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.245088    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-serving-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:02.245125 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.245118    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t657z\" (UniqueName: \"kubernetes.io/projected/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-kube-api-access-t657z\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:02.245125 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.245128    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e88b26-ca27-45cf-83c2-5a84bb0c7e82-console-oauth-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:02.790130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.790088    2581 generic.go:356] "Generic (PLEG): container finished" podID="87e88b26-ca27-45cf-83c2-5a84bb0c7e82" containerID="959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84" exitCode=0
Sep 16 00:51:02.790661 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.790135    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66f48fc8-zg4m9"
Sep 16 00:51:02.790661 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.790177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66f48fc8-zg4m9" event={"ID":"87e88b26-ca27-45cf-83c2-5a84bb0c7e82","Type":"ContainerDied","Data":"959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84"}
Sep 16 00:51:02.790661 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.790233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66f48fc8-zg4m9" event={"ID":"87e88b26-ca27-45cf-83c2-5a84bb0c7e82","Type":"ContainerDied","Data":"131a475c5fe72030d77f09104d7414576cb2bd5f5ac0876de41e4c186eae6b26"}
Sep 16 00:51:02.790661 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.790255    2581 scope.go:117] "RemoveContainer" containerID="959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84"
Sep 16 00:51:02.798779 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.798763    2581 scope.go:117] "RemoveContainer" containerID="959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84"
Sep 16 00:51:02.799035 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:51:02.799016    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84\": container with ID starting with 959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84 not found: ID does not exist" containerID="959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84"
Sep 16 00:51:02.799079 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.799045    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84"} err="failed to get container status \"959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84\": rpc error: code = NotFound desc = could not find container \"959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84\": container with ID starting with 959440ad24c93766f9c33e69dda44f7cc48db1aee09b5d093e67cdf900a0bf84 not found: ID does not exist"
Sep 16 00:51:02.812212 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.812179    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d66f48fc8-zg4m9"]
Sep 16 00:51:02.815328 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:02.815301    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d66f48fc8-zg4m9"]
Sep 16 00:51:03.940766 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:03.940730    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e88b26-ca27-45cf-83c2-5a84bb0c7e82" path="/var/lib/kubelet/pods/87e88b26-ca27-45cf-83c2-5a84bb0c7e82/volumes"
Sep 16 00:51:07.445741 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:07.445680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:07.545674 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:07.545644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:08.120696 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.120660    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"]
Sep 16 00:51:08.120999 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.120987    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e88b26-ca27-45cf-83c2-5a84bb0c7e82" containerName="console"
Sep 16 00:51:08.121043 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.121001    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e88b26-ca27-45cf-83c2-5a84bb0c7e82" containerName="console"
Sep 16 00:51:08.121078 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.121073    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e88b26-ca27-45cf-83c2-5a84bb0c7e82" containerName="console"
Sep 16 00:51:08.123875 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.123856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.126191 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.126168    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 00:51:08.127039 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.127015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t4f99\""
Sep 16 00:51:08.127154 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.127094    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 00:51:08.132369 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.132343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"]
Sep 16 00:51:08.190052 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.190015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.190052 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.190059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.190290 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.190130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4js5x\" (UniqueName: \"kubernetes.io/projected/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-kube-api-access-4js5x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.290559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.290517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.290559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.290565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.290782 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.290607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4js5x\" (UniqueName: \"kubernetes.io/projected/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-kube-api-access-4js5x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.290972 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.290948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.291031 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.290966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.298393 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.298369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4js5x\" (UniqueName: \"kubernetes.io/projected/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-kube-api-access-4js5x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.434165 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.434081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:08.701689 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.701624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"]
Sep 16 00:51:08.704086 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:51:08.704053    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3acfc41_f15d_42bc_8b7c_b8d250ca7cc6.slice/crio-1388a70525ece609dcaa92f6ce0afad37a752a2b63882f7feafa29d1ed33fa57 WatchSource:0}: Error finding container 1388a70525ece609dcaa92f6ce0afad37a752a2b63882f7feafa29d1ed33fa57: Status 404 returned error can't find the container with id 1388a70525ece609dcaa92f6ce0afad37a752a2b63882f7feafa29d1ed33fa57
Sep 16 00:51:08.816144 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:08.816105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh" event={"ID":"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6","Type":"ContainerStarted","Data":"1388a70525ece609dcaa92f6ce0afad37a752a2b63882f7feafa29d1ed33fa57"}
Sep 16 00:51:15.854059 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:15.854027    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerID="1ecdab1845f3268863c3ccf09762a119ce5d68217cf7644c503b93e506979592" exitCode=0
Sep 16 00:51:15.854464 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:15.854118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh" event={"ID":"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6","Type":"ContainerDied","Data":"1ecdab1845f3268863c3ccf09762a119ce5d68217cf7644c503b93e506979592"}
Sep 16 00:51:18.868806 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:18.868765    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerID="b39d5a699627a6dcdc7e8447911cb7c785a78ac8c361c59fdf5650b29bd60fd7" exitCode=0
Sep 16 00:51:18.869277 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:18.868851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh" event={"ID":"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6","Type":"ContainerDied","Data":"b39d5a699627a6dcdc7e8447911cb7c785a78ac8c361c59fdf5650b29bd60fd7"}
Sep 16 00:51:25.905158 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:25.905124    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerID="f0ea473449ae9727c437522370f22eb492d1ce6e62c04b93c738cf49fdb3c813" exitCode=0
Sep 16 00:51:25.905556 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:25.905192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh" event={"ID":"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6","Type":"ContainerDied","Data":"f0ea473449ae9727c437522370f22eb492d1ce6e62c04b93c738cf49fdb3c813"}
Sep 16 00:51:27.036735 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.036707    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:27.160965 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.160861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4js5x\" (UniqueName: \"kubernetes.io/projected/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-kube-api-access-4js5x\") pod \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") "
Sep 16 00:51:27.160965 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.160938    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-bundle\") pod \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") "
Sep 16 00:51:27.161203 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.161001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-util\") pod \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\" (UID: \"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6\") "
Sep 16 00:51:27.161669 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.161644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-bundle" (OuterVolumeSpecName: "bundle") pod "e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" (UID: "e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:51:27.163182 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.163157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-kube-api-access-4js5x" (OuterVolumeSpecName: "kube-api-access-4js5x") pod "e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" (UID: "e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6"). InnerVolumeSpecName "kube-api-access-4js5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:51:27.166055 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.166029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-util" (OuterVolumeSpecName: "util") pod "e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" (UID: "e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:51:27.262454 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.262411    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-bundle\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:27.262454 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.262447    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-util\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:27.262454 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.262460    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4js5x\" (UniqueName: \"kubernetes.io/projected/e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6-kube-api-access-4js5x\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:51:27.915464 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.915426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh" event={"ID":"e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6","Type":"ContainerDied","Data":"1388a70525ece609dcaa92f6ce0afad37a752a2b63882f7feafa29d1ed33fa57"}
Sep 16 00:51:27.915464 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.915457    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540whzrh"
Sep 16 00:51:27.915739 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:27.915474    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1388a70525ece609dcaa92f6ce0afad37a752a2b63882f7feafa29d1ed33fa57"
Sep 16 00:51:29.140146 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:29.140112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:29.886382 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:29.886346    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:31.042032 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.041998    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"]
Sep 16 00:51:31.042418 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.042326    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerName="util"
Sep 16 00:51:31.042418 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.042337    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerName="util"
Sep 16 00:51:31.042418 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.042345    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerName="pull"
Sep 16 00:51:31.042418 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.042350    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerName="pull"
Sep 16 00:51:31.042418 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.042359    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerName="extract"
Sep 16 00:51:31.042418 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.042365    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerName="extract"
Sep 16 00:51:31.042418 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.042416    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3acfc41-f15d-42bc-8b7c-b8d250ca7cc6" containerName="extract"
Sep 16 00:51:31.046130 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.046113    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.048470 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.048447    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 00:51:31.048659 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.048640    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:51:31.048701 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.048660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6fmks\""
Sep 16 00:51:31.057541 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.057516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"]
Sep 16 00:51:31.097895 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.097851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b82d8816-a7db-47e8-9b99-2a3b7d23bec9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vrsgw\" (UID: \"b82d8816-a7db-47e8-9b99-2a3b7d23bec9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.098082 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.097907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpvs\" (UniqueName: \"kubernetes.io/projected/b82d8816-a7db-47e8-9b99-2a3b7d23bec9-kube-api-access-srpvs\") pod \"cert-manager-operator-controller-manager-55c444bf86-vrsgw\" (UID: \"b82d8816-a7db-47e8-9b99-2a3b7d23bec9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.198448 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.198402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b82d8816-a7db-47e8-9b99-2a3b7d23bec9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vrsgw\" (UID: \"b82d8816-a7db-47e8-9b99-2a3b7d23bec9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.198646 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.198456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpvs\" (UniqueName: \"kubernetes.io/projected/b82d8816-a7db-47e8-9b99-2a3b7d23bec9-kube-api-access-srpvs\") pod \"cert-manager-operator-controller-manager-55c444bf86-vrsgw\" (UID: \"b82d8816-a7db-47e8-9b99-2a3b7d23bec9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.198898 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.198878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b82d8816-a7db-47e8-9b99-2a3b7d23bec9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vrsgw\" (UID: \"b82d8816-a7db-47e8-9b99-2a3b7d23bec9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.206810 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.206786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpvs\" (UniqueName: \"kubernetes.io/projected/b82d8816-a7db-47e8-9b99-2a3b7d23bec9-kube-api-access-srpvs\") pod \"cert-manager-operator-controller-manager-55c444bf86-vrsgw\" (UID: \"b82d8816-a7db-47e8-9b99-2a3b7d23bec9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.355792 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.355705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"
Sep 16 00:51:31.628167 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.628142    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw"]
Sep 16 00:51:31.631098 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:51:31.631060    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82d8816_a7db_47e8_9b99_2a3b7d23bec9.slice/crio-dab4d34afb8b1a483554e0d7e3553291d962a530691367011b1b2eb99e594ca1 WatchSource:0}: Error finding container dab4d34afb8b1a483554e0d7e3553291d962a530691367011b1b2eb99e594ca1: Status 404 returned error can't find the container with id dab4d34afb8b1a483554e0d7e3553291d962a530691367011b1b2eb99e594ca1
Sep 16 00:51:31.933526 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:31.933434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw" event={"ID":"b82d8816-a7db-47e8-9b99-2a3b7d23bec9","Type":"ContainerStarted","Data":"dab4d34afb8b1a483554e0d7e3553291d962a530691367011b1b2eb99e594ca1"}
Sep 16 00:51:32.718223 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:32.718190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:32.817920 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:32.817885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:33.943544 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:33.943512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw" event={"ID":"b82d8816-a7db-47e8-9b99-2a3b7d23bec9","Type":"ContainerStarted","Data":"4132941baadb1bbc6b547fb42bad12317d2252d2d79fdf87435a299a3ddbd2de"}
Sep 16 00:51:33.963628 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:33.963570    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vrsgw" podStartSLOduration=1.23745008 podStartE2EDuration="2.963554525s" totalImagesPullingTime="1.726104445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:33.961334217 +0000 UTC m=+512.504471193" watchObservedRunningTime="2026-09-16 00:51:33.963554525 +0000 UTC m=+512.506691499"
Sep 16 00:51:34.301678 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.301641    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:34.393674 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.393638    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:34.529416 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.529375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:34.646895 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.646796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:34.679276 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.679246    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-drhlq"]
Sep 16 00:51:34.681517 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.681474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:34.683840 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.683821    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 00:51:34.684746 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.684721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7zm9g\""
Sep 16 00:51:34.684833 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.684729    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 00:51:34.692566 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.692543    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-drhlq"]
Sep 16 00:51:34.835598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.835545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38a0dde3-17b9-4419-8c54-2b1847c071da-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-drhlq\" (UID: \"38a0dde3-17b9-4419-8c54-2b1847c071da\") " pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:34.835783 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.835669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5826x\" (UniqueName: \"kubernetes.io/projected/38a0dde3-17b9-4419-8c54-2b1847c071da-kube-api-access-5826x\") pod \"cert-manager-webhook-6486ffd997-drhlq\" (UID: \"38a0dde3-17b9-4419-8c54-2b1847c071da\") " pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:34.936471 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.936387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5826x\" (UniqueName: \"kubernetes.io/projected/38a0dde3-17b9-4419-8c54-2b1847c071da-kube-api-access-5826x\") pod \"cert-manager-webhook-6486ffd997-drhlq\" (UID: \"38a0dde3-17b9-4419-8c54-2b1847c071da\") " pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:34.936471 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.936449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38a0dde3-17b9-4419-8c54-2b1847c071da-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-drhlq\" (UID: \"38a0dde3-17b9-4419-8c54-2b1847c071da\") " pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:34.944284 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.944259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5826x\" (UniqueName: \"kubernetes.io/projected/38a0dde3-17b9-4419-8c54-2b1847c071da-kube-api-access-5826x\") pod \"cert-manager-webhook-6486ffd997-drhlq\" (UID: \"38a0dde3-17b9-4419-8c54-2b1847c071da\") " pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:34.944613 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:34.944284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38a0dde3-17b9-4419-8c54-2b1847c071da-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-drhlq\" (UID: \"38a0dde3-17b9-4419-8c54-2b1847c071da\") " pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:35.000377 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:35.000342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:35.268124 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:35.268097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-drhlq"]
Sep 16 00:51:35.270425 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:51:35.270402    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38a0dde3_17b9_4419_8c54_2b1847c071da.slice/crio-6bc8cdf9137c4b4c8f9f62cab54915ae7cafe77104b55f25fc43369cc782e745 WatchSource:0}: Error finding container 6bc8cdf9137c4b4c8f9f62cab54915ae7cafe77104b55f25fc43369cc782e745: Status 404 returned error can't find the container with id 6bc8cdf9137c4b4c8f9f62cab54915ae7cafe77104b55f25fc43369cc782e745
Sep 16 00:51:35.952561 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:35.952518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq" event={"ID":"38a0dde3-17b9-4419-8c54-2b1847c071da","Type":"ContainerStarted","Data":"6bc8cdf9137c4b4c8f9f62cab54915ae7cafe77104b55f25fc43369cc782e745"}
Sep 16 00:51:36.090641 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:36.090603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:36.692693 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:36.692630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:36.918835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:36.918796    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-999sh"]
Sep 16 00:51:36.921565 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:36.921539    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:36.923598 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:36.923567    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5wthr\""
Sep 16 00:51:36.932372 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:36.932336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-999sh"]
Sep 16 00:51:37.056773 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.056740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6f140a-b7e1-4b4d-9b6e-27c82bed5810-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-999sh\" (UID: \"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:37.057221 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.056791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzrw\" (UniqueName: \"kubernetes.io/projected/ab6f140a-b7e1-4b4d-9b6e-27c82bed5810-kube-api-access-fdzrw\") pod \"cert-manager-cainjector-559cdd8c87-999sh\" (UID: \"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:37.158177 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.158136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6f140a-b7e1-4b4d-9b6e-27c82bed5810-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-999sh\" (UID: \"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:37.158378 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.158192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzrw\" (UniqueName: \"kubernetes.io/projected/ab6f140a-b7e1-4b4d-9b6e-27c82bed5810-kube-api-access-fdzrw\") pod \"cert-manager-cainjector-559cdd8c87-999sh\" (UID: \"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:37.168464 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.168424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6f140a-b7e1-4b4d-9b6e-27c82bed5810-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-999sh\" (UID: \"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:37.168625 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.168572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzrw\" (UniqueName: \"kubernetes.io/projected/ab6f140a-b7e1-4b4d-9b6e-27c82bed5810-kube-api-access-fdzrw\") pod \"cert-manager-cainjector-559cdd8c87-999sh\" (UID: \"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:37.233633 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.233597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh"
Sep 16 00:51:37.964227 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.964193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq" event={"ID":"38a0dde3-17b9-4419-8c54-2b1847c071da","Type":"ContainerStarted","Data":"73c82ab76c5d4f84400f01f232ef29433114169ca0cb2543047d24017d923eeb"}
Sep 16 00:51:37.964407 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.964247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:37.981951 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:37.981888    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq" podStartSLOduration=1.3773381900000001 podStartE2EDuration="3.981869836s" totalImagesPullingTime="2.604531646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:37.979266891 +0000 UTC m=+516.522403867" watchObservedRunningTime="2026-09-16 00:51:37.981869836 +0000 UTC m=+516.525006812"
Sep 16 00:51:38.102669 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:38.102638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-999sh"]
Sep 16 00:51:38.104950 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:51:38.104924    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab6f140a_b7e1_4b4d_9b6e_27c82bed5810.slice/crio-093d1300990609b81a07e2a4d5a51108ab67645cfbe7545adf8634186de91e6e WatchSource:0}: Error finding container 093d1300990609b81a07e2a4d5a51108ab67645cfbe7545adf8634186de91e6e: Status 404 returned error can't find the container with id 093d1300990609b81a07e2a4d5a51108ab67645cfbe7545adf8634186de91e6e
Sep 16 00:51:38.970083 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:38.970044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh" event={"ID":"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810","Type":"ContainerStarted","Data":"7a3da20487457bb1c7c5f6a88bbcdf22b99c4f2ac330b4f284c1878cde78d5f7"}
Sep 16 00:51:38.970083 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:38.970088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh" event={"ID":"ab6f140a-b7e1-4b4d-9b6e-27c82bed5810","Type":"ContainerStarted","Data":"093d1300990609b81a07e2a4d5a51108ab67645cfbe7545adf8634186de91e6e"}
Sep 16 00:51:38.984201 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:38.984154    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-999sh" podStartSLOduration=2.984138761 podStartE2EDuration="2.984138761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:38.982944515 +0000 UTC m=+517.526081490" watchObservedRunningTime="2026-09-16 00:51:38.984138761 +0000 UTC m=+517.527275736"
Sep 16 00:51:39.503096 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:39.503064    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:39.557061 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:39.557032    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:43.972662 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:43.972631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-drhlq"
Sep 16 00:51:45.487400 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:45.487360    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:46.198016 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.197978    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gb6z2"]
Sep 16 00:51:46.201644 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.201617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.204011 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.203985    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-67lnh\""
Sep 16 00:51:46.211038 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.211007    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gb6z2"]
Sep 16 00:51:46.337119 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.337081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac0348c-dc9c-41a0-812d-4c7be50bd0e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-gb6z2\" (UID: \"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0\") " pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.337119 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.337124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rdc\" (UniqueName: \"kubernetes.io/projected/3ac0348c-dc9c-41a0-812d-4c7be50bd0e0-kube-api-access-g6rdc\") pod \"cert-manager-55b4d954d5-gb6z2\" (UID: \"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0\") " pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.437976 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.437936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac0348c-dc9c-41a0-812d-4c7be50bd0e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-gb6z2\" (UID: \"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0\") " pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.437976 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.437983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rdc\" (UniqueName: \"kubernetes.io/projected/3ac0348c-dc9c-41a0-812d-4c7be50bd0e0-kube-api-access-g6rdc\") pod \"cert-manager-55b4d954d5-gb6z2\" (UID: \"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0\") " pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.445388 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.445349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac0348c-dc9c-41a0-812d-4c7be50bd0e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-gb6z2\" (UID: \"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0\") " pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.445562 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.445422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rdc\" (UniqueName: \"kubernetes.io/projected/3ac0348c-dc9c-41a0-812d-4c7be50bd0e0-kube-api-access-g6rdc\") pod \"cert-manager-55b4d954d5-gb6z2\" (UID: \"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0\") " pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.512442 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.512413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gb6z2"
Sep 16 00:51:46.780415 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:46.780389    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gb6z2"]
Sep 16 00:51:46.783283 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:51:46.783257    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ac0348c_dc9c_41a0_812d_4c7be50bd0e0.slice/crio-9ed56d2854006a72a4c419d580646b596224c61273f7cfe5b1133db3709325f2 WatchSource:0}: Error finding container 9ed56d2854006a72a4c419d580646b596224c61273f7cfe5b1133db3709325f2: Status 404 returned error can't find the container with id 9ed56d2854006a72a4c419d580646b596224c61273f7cfe5b1133db3709325f2
Sep 16 00:51:47.007613 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:47.007580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gb6z2" event={"ID":"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0","Type":"ContainerStarted","Data":"20c06fd99b8a2ade5c339e286a009c06677d14f418460f53cb2aa1ef3cfe0c7e"}
Sep 16 00:51:47.007785 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:47.007620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gb6z2" event={"ID":"3ac0348c-dc9c-41a0-812d-4c7be50bd0e0","Type":"ContainerStarted","Data":"9ed56d2854006a72a4c419d580646b596224c61273f7cfe5b1133db3709325f2"}
Sep 16 00:51:47.034030 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:47.033924    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gb6z2" podStartSLOduration=1.03390522 podStartE2EDuration="1.03390522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:47.031246128 +0000 UTC m=+525.574383103" watchObservedRunningTime="2026-09-16 00:51:47.03390522 +0000 UTC m=+525.577042195"
Sep 16 00:51:53.276886 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:53.276854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:53.338930 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:53.338898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:57.239841 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:57.239753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:57.339123 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:57.339088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:58.215204 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:58.215169    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:51:58.270835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:51:58.270802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:03.377677 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:03.377639    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:03.491367 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:03.491323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:04.989861 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:04.989827    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"]
Sep 16 00:52:04.997554 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:04.997523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:04.999772 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:04.999747    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 00:52:04.999772 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:04.999766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 00:52:04.999949 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:04.999773    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t4f99\""
Sep 16 00:52:05.002876 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.002850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"]
Sep 16 00:52:05.097523 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.097472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.097715 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.097577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.097715 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.097610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvfh\" (UniqueName: \"kubernetes.io/projected/f7d10776-2281-45b5-990c-bb3546691a2d-kube-api-access-lzvfh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.109821 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.109792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:05.198462 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.198422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvfh\" (UniqueName: \"kubernetes.io/projected/f7d10776-2281-45b5-990c-bb3546691a2d-kube-api-access-lzvfh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.198675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.198560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.198675 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.198599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.198947 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.198921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.199015 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.198950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.207057 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.207030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvfh\" (UniqueName: \"kubernetes.io/projected/f7d10776-2281-45b5-990c-bb3546691a2d-kube-api-access-lzvfh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.307835 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.307800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:05.577620 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:05.577597    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"]
Sep 16 00:52:05.579876 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:52:05.579849    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d10776_2281_45b5_990c_bb3546691a2d.slice/crio-0850108f0212ab92d6f2a11e82f34926abc476c78d1f67c62ece468e92a4ec70 WatchSource:0}: Error finding container 0850108f0212ab92d6f2a11e82f34926abc476c78d1f67c62ece468e92a4ec70: Status 404 returned error can't find the container with id 0850108f0212ab92d6f2a11e82f34926abc476c78d1f67c62ece468e92a4ec70
Sep 16 00:52:06.095921 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:06.095885    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7d10776-2281-45b5-990c-bb3546691a2d" containerID="48789d8514394900989969c6cc655f06c6f5c234eaf16960a609d0007b8d5791" exitCode=0
Sep 16 00:52:06.096279 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:06.095968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt" event={"ID":"f7d10776-2281-45b5-990c-bb3546691a2d","Type":"ContainerDied","Data":"48789d8514394900989969c6cc655f06c6f5c234eaf16960a609d0007b8d5791"}
Sep 16 00:52:06.096279 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:06.096012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt" event={"ID":"f7d10776-2281-45b5-990c-bb3546691a2d","Type":"ContainerStarted","Data":"0850108f0212ab92d6f2a11e82f34926abc476c78d1f67c62ece468e92a4ec70"}
Sep 16 00:52:07.102713 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:07.102676    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7d10776-2281-45b5-990c-bb3546691a2d" containerID="3d2518f260da3da687024763712ce98d72c1ab063b19eff3ede3deb89f4c126f" exitCode=0
Sep 16 00:52:07.103131 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:07.102731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt" event={"ID":"f7d10776-2281-45b5-990c-bb3546691a2d","Type":"ContainerDied","Data":"3d2518f260da3da687024763712ce98d72c1ab063b19eff3ede3deb89f4c126f"}
Sep 16 00:52:08.108863 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:08.108829    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7d10776-2281-45b5-990c-bb3546691a2d" containerID="ecfaeee613b1ee5e927b32627ff4f30d0e1f55932707a47eae7f0b74424daed3" exitCode=0
Sep 16 00:52:08.109257 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:08.108901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt" event={"ID":"f7d10776-2281-45b5-990c-bb3546691a2d","Type":"ContainerDied","Data":"ecfaeee613b1ee5e927b32627ff4f30d0e1f55932707a47eae7f0b74424daed3"}
Sep 16 00:52:09.241601 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.241576    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:09.334808 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.334775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-bundle\") pod \"f7d10776-2281-45b5-990c-bb3546691a2d\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") "
Sep 16 00:52:09.334808 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.334822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-util\") pod \"f7d10776-2281-45b5-990c-bb3546691a2d\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") "
Sep 16 00:52:09.335085 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.334851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzvfh\" (UniqueName: \"kubernetes.io/projected/f7d10776-2281-45b5-990c-bb3546691a2d-kube-api-access-lzvfh\") pod \"f7d10776-2281-45b5-990c-bb3546691a2d\" (UID: \"f7d10776-2281-45b5-990c-bb3546691a2d\") "
Sep 16 00:52:09.335749 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.335708    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-bundle" (OuterVolumeSpecName: "bundle") pod "f7d10776-2281-45b5-990c-bb3546691a2d" (UID: "f7d10776-2281-45b5-990c-bb3546691a2d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:52:09.336979 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.336955    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d10776-2281-45b5-990c-bb3546691a2d-kube-api-access-lzvfh" (OuterVolumeSpecName: "kube-api-access-lzvfh") pod "f7d10776-2281-45b5-990c-bb3546691a2d" (UID: "f7d10776-2281-45b5-990c-bb3546691a2d"). InnerVolumeSpecName "kube-api-access-lzvfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:52:09.340550 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.340528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-util" (OuterVolumeSpecName: "util") pod "f7d10776-2281-45b5-990c-bb3546691a2d" (UID: "f7d10776-2281-45b5-990c-bb3546691a2d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:52:09.435727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.435632    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-util\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:52:09.435727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.435668    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvfh\" (UniqueName: \"kubernetes.io/projected/f7d10776-2281-45b5-990c-bb3546691a2d-kube-api-access-lzvfh\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:52:09.435727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:09.435678    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7d10776-2281-45b5-990c-bb3546691a2d-bundle\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:52:10.119423 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:10.119388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt" event={"ID":"f7d10776-2281-45b5-990c-bb3546691a2d","Type":"ContainerDied","Data":"0850108f0212ab92d6f2a11e82f34926abc476c78d1f67c62ece468e92a4ec70"}
Sep 16 00:52:10.119423 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:10.119424    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznhbt"
Sep 16 00:52:10.119657 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:10.119434    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0850108f0212ab92d6f2a11e82f34926abc476c78d1f67c62ece468e92a4ec70"
Sep 16 00:52:13.943334 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:13.943281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:19.262788 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.262753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:19.329625 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.329584    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"]
Sep 16 00:52:19.330018 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.330000    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d10776-2281-45b5-990c-bb3546691a2d" containerName="pull"
Sep 16 00:52:19.330102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.330021    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d10776-2281-45b5-990c-bb3546691a2d" containerName="pull"
Sep 16 00:52:19.330102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.330039    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d10776-2281-45b5-990c-bb3546691a2d" containerName="util"
Sep 16 00:52:19.330102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.330046    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d10776-2281-45b5-990c-bb3546691a2d" containerName="util"
Sep 16 00:52:19.330102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.330061    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d10776-2281-45b5-990c-bb3546691a2d" containerName="extract"
Sep 16 00:52:19.330102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.330071    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d10776-2281-45b5-990c-bb3546691a2d" containerName="extract"
Sep 16 00:52:19.330401 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.330179    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d10776-2281-45b5-990c-bb3546691a2d" containerName="extract"
Sep 16 00:52:19.334692 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.334670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.339242 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.339220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 00:52:19.340123 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.340103    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:52:19.340224 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.340147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-79rlj\""
Sep 16 00:52:19.340324 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.340304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 00:52:19.341318 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.340968    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 00:52:19.341318 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.340974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 00:52:19.353852 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.353825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"]
Sep 16 00:52:19.372388 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.372359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:19.425724 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.425689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-cert\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.425886 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.425745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-manager-config\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.425886 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.425813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-metrics-cert\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.425886 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.425840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g456t\" (UniqueName: \"kubernetes.io/projected/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-kube-api-access-g456t\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.526830 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.526733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-metrics-cert\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.526830 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.526771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g456t\" (UniqueName: \"kubernetes.io/projected/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-kube-api-access-g456t\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.526830 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.526809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-cert\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.527122 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.526840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-manager-config\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.527515 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.527471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-manager-config\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.529526 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.529501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-metrics-cert\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.529526 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.529517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-cert\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.534605 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.534582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g456t\" (UniqueName: \"kubernetes.io/projected/a83eef67-8cdc-4413-9cd3-3732a1f00ba4-kube-api-access-g456t\") pod \"lws-controller-manager-54b79494c6-wbhdz\" (UID: \"a83eef67-8cdc-4413-9cd3-3732a1f00ba4\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.643825 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.643786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:19.915591 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:19.915565    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"]
Sep 16 00:52:19.917710 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:52:19.917679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83eef67_8cdc_4413_9cd3_3732a1f00ba4.slice/crio-12fd15eb3e018f6175d8ba9b357b83fc58e2448ae13f79b28f3212445aed731f WatchSource:0}: Error finding container 12fd15eb3e018f6175d8ba9b357b83fc58e2448ae13f79b28f3212445aed731f: Status 404 returned error can't find the container with id 12fd15eb3e018f6175d8ba9b357b83fc58e2448ae13f79b28f3212445aed731f
Sep 16 00:52:20.167080 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:20.166991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz" event={"ID":"a83eef67-8cdc-4413-9cd3-3732a1f00ba4","Type":"ContainerStarted","Data":"12fd15eb3e018f6175d8ba9b357b83fc58e2448ae13f79b28f3212445aed731f"}
Sep 16 00:52:21.429294 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:21.429242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:21.521697 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:21.521632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.055858 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.055824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.238782 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.238750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.337441 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.337348    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.430187 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.430159    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"]
Sep 16 00:52:22.434745 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.434727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.437545 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.437470    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 00:52:22.437545 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.437521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 00:52:22.437545 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.437522    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cqdqc\""
Sep 16 00:52:22.437767 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.437551    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 00:52:22.438089 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.437850    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 00:52:22.446842 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.446816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"]
Sep 16 00:52:22.542342 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.542303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.556649 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.556615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a068eb0-afe6-4138-8813-c8f2164881c4-tmp\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.556789 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.556658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a068eb0-afe6-4138-8813-c8f2164881c4-manifests\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.556789 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.556679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a068eb0-afe6-4138-8813-c8f2164881c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.556789 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.556744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a068eb0-afe6-4138-8813-c8f2164881c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.556925 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.556855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttm5c\" (UniqueName: \"kubernetes.io/projected/2a068eb0-afe6-4138-8813-c8f2164881c4-kube-api-access-ttm5c\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.642570 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.642440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.657713 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.657679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttm5c\" (UniqueName: \"kubernetes.io/projected/2a068eb0-afe6-4138-8813-c8f2164881c4-kube-api-access-ttm5c\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.657892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.657737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a068eb0-afe6-4138-8813-c8f2164881c4-tmp\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.657892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.657772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a068eb0-afe6-4138-8813-c8f2164881c4-manifests\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.657892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.657797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a068eb0-afe6-4138-8813-c8f2164881c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.657892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.657836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a068eb0-afe6-4138-8813-c8f2164881c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.658235 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.658208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a068eb0-afe6-4138-8813-c8f2164881c4-manifests\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.660513 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.660470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a068eb0-afe6-4138-8813-c8f2164881c4-tmp\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.660975 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.660953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a068eb0-afe6-4138-8813-c8f2164881c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.661092 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.661073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a068eb0-afe6-4138-8813-c8f2164881c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.673962 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.673934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttm5c\" (UniqueName: \"kubernetes.io/projected/2a068eb0-afe6-4138-8813-c8f2164881c4-kube-api-access-ttm5c\") pod \"opendatahub-operator-controller-manager-9778fc5bd-x56vf\" (UID: \"2a068eb0-afe6-4138-8813-c8f2164881c4\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.745901 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.745866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.750815 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.750793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:22.842861 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.842826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:22.942894 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:22.942800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:23.034331 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:23.034260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"]
Sep 16 00:52:23.037415 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:52:23.037382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a068eb0_afe6_4138_8813_c8f2164881c4.slice/crio-2cedf3a7b1a3a4b767ba754fe0287c99a7f0fba793441ebb932fd0f967a8708b WatchSource:0}: Error finding container 2cedf3a7b1a3a4b767ba754fe0287c99a7f0fba793441ebb932fd0f967a8708b: Status 404 returned error can't find the container with id 2cedf3a7b1a3a4b767ba754fe0287c99a7f0fba793441ebb932fd0f967a8708b
Sep 16 00:52:23.037547 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:23.037528    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:23.184704 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:23.184662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz" event={"ID":"a83eef67-8cdc-4413-9cd3-3732a1f00ba4","Type":"ContainerStarted","Data":"c8f032afe490694845bb375a46bfe6b54467ab20a071c8edd46819c6f1e6d348"}
Sep 16 00:52:23.184902 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:23.184787    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:23.185966 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:23.185941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf" event={"ID":"2a068eb0-afe6-4138-8813-c8f2164881c4","Type":"ContainerStarted","Data":"2cedf3a7b1a3a4b767ba754fe0287c99a7f0fba793441ebb932fd0f967a8708b"}
Sep 16 00:52:23.200602 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:23.200505    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz" podStartSLOduration=1.7190757269999999 podStartE2EDuration="4.200473926s" totalImagesPullingTime="2.481398199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:23.198893462 +0000 UTC m=+561.742030436" watchObservedRunningTime="2026-09-16 00:52:23.200473926 +0000 UTC m=+561.743610966"
Sep 16 00:52:26.203318 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:26.203280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf" event={"ID":"2a068eb0-afe6-4138-8813-c8f2164881c4","Type":"ContainerStarted","Data":"394c141824788bd0db988dc82b62c55f003a8fbdf6a9bbe59671c81e90d40c51"}
Sep 16 00:52:27.209617 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:27.209579    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a068eb0-afe6-4138-8813-c8f2164881c4" containerID="394c141824788bd0db988dc82b62c55f003a8fbdf6a9bbe59671c81e90d40c51" exitCode=0
Sep 16 00:52:27.210025 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:27.209660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf" event={"ID":"2a068eb0-afe6-4138-8813-c8f2164881c4","Type":"ContainerDied","Data":"394c141824788bd0db988dc82b62c55f003a8fbdf6a9bbe59671c81e90d40c51"}
Sep 16 00:52:28.216721 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:28.216684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf" event={"ID":"2a068eb0-afe6-4138-8813-c8f2164881c4","Type":"ContainerStarted","Data":"101f422c7fc69e3bb0dd3c39abd9b4c35eb53ce121d14b5757f2e66ea2a915b9"}
Sep 16 00:52:28.217128 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:28.216799    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:28.238191 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:28.238139    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf" podStartSLOduration=1.86093192 podStartE2EDuration="6.23812064s" totalImagesPullingTime="3.346752337s" totalInitContainerRuntime="1.030436383s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:28.236377819 +0000 UTC m=+566.779514835" watchObservedRunningTime="2026-09-16 00:52:28.23812064 +0000 UTC m=+566.781257614"
Sep 16 00:52:34.194521 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:34.194468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-wbhdz"
Sep 16 00:52:34.643759 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:34.643726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:35.949781 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:35.949748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:39.225065 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:39.225035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-x56vf"
Sep 16 00:52:41.214152 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:41.214112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:41.315663 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:41.315629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:41.589090 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:41.589055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:41.713716 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:41.713679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:41.940232 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:41.940148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:41.989559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:41.989522    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"]
Sep 16 00:52:42.008461 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.008429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.012646 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.012618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 00:52:42.013065 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.013048    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v2b4x\""
Sep 16 00:52:42.013399 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.013384    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 00:52:42.013631 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.013604    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 00:52:42.013804 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.013790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 00:52:42.014679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.014631    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"]
Sep 16 00:52:42.046524 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.046473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tmp\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.046685 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.046559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tls-certs\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.046685 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.046612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znp6b\" (UniqueName: \"kubernetes.io/projected/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-kube-api-access-znp6b\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.147999 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.147960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tmp\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.148194 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.148025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tls-certs\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.148194 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.148060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znp6b\" (UniqueName: \"kubernetes.io/projected/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-kube-api-access-znp6b\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.148194 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:52:42.148165    2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 00:52:42.148354 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:52:42.148252    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tls-certs podName:c6fb763b-3b45-44b3-a733-3ac5f23ed8c9 nodeName:}" failed. No retries permitted until 2026-09-16 00:52:42.648230602 +0000 UTC m=+581.191367560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tls-certs") pod "kube-auth-proxy-5bfdbc6f8d-xj5xh" (UID: "c6fb763b-3b45-44b3-a733-3ac5f23ed8c9") : secret "kube-auth-proxy-tls" not found
Sep 16 00:52:42.150236 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.150216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tmp\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.168224 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.168191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znp6b\" (UniqueName: \"kubernetes.io/projected/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-kube-api-access-znp6b\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.493249 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.493214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:42.651950 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.651908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tls-certs\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.654438 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.654413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fb763b-3b45-44b3-a733-3ac5f23ed8c9-tls-certs\") pod \"kube-auth-proxy-5bfdbc6f8d-xj5xh\" (UID: \"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9\") " pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:42.923271 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:42.923229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"
Sep 16 00:52:43.196724 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:43.196698    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh"]
Sep 16 00:52:43.199495 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:52:43.199459    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fb763b_3b45_44b3_a733_3ac5f23ed8c9.slice/crio-ca964cf6ee30edc923750d76746ac82b542edd3b1b5d9aef16549f175c44b28f WatchSource:0}: Error finding container ca964cf6ee30edc923750d76746ac82b542edd3b1b5d9aef16549f175c44b28f: Status 404 returned error can't find the container with id ca964cf6ee30edc923750d76746ac82b542edd3b1b5d9aef16549f175c44b28f
Sep 16 00:52:43.293776 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:43.293739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh" event={"ID":"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9","Type":"ContainerStarted","Data":"ca964cf6ee30edc923750d76746ac82b542edd3b1b5d9aef16549f175c44b28f"}
Sep 16 00:52:43.384625 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:43.384589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:43.449794 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:43.449711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:44.316113 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:44.316073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:46.312869 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:46.312828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh" event={"ID":"c6fb763b-3b45-44b3-a733-3ac5f23ed8c9","Type":"ContainerStarted","Data":"7b513e9fde808f2107edd5028f7842d67b36fd151a7505319e703995e4cba76d"}
Sep 16 00:52:46.330031 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:46.329974    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bfdbc6f8d-xj5xh" podStartSLOduration=2.5575488589999997 podStartE2EDuration="5.329958459s" totalImagesPullingTime="2.7724096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:46.328441895 +0000 UTC m=+584.871578870" watchObservedRunningTime="2026-09-16 00:52:46.329958459 +0000 UTC m=+584.873095459"
Sep 16 00:52:48.556993 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:48.556959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:48.657642 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:48.657608    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:49.143859 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:49.143826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:49.337121 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:49.337071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:49.528626 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:49.528596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:52.688197 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:52.688158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:52.759344 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:52.759300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:52.860870 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:52.860829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:52.964271 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:52.964177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:53.330176 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:53.330134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:52:53.489178 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:52:53.489119    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:01.871808 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:01.871769    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:53:01.872247 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:01.871859    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:53:02.028505 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:02.028446    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:02.133173 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:02.133079    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:02.229596 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:02.229558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:02.492734 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:02.492661    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:03.312110 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:03.312078    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:03.390823 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:03.390791    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:20.554762 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:20.554719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:20.607829 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:20.607788    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:20.692980 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:20.692948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:20.749030 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:20.748986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:21.305117 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:21.305075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:21.369620 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:21.369583    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:31.053814 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:31.053733    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567bdd56d6-727qs"]
Sep 16 00:53:31.157560 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:31.157529    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:31.260533 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:31.260500    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:31.359714 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:31.359615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:53.827659 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:53.827615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:53.911113 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:53.911081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:53:56.073993 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.073934    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-567bdd56d6-727qs" podUID="4b3b6fd3-450f-4b2e-bc08-219245bcc83c" containerName="console" containerID="cri-o://76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51" gracePeriod=15
Sep 16 00:53:56.330890 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.330828    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:53:56.425075 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425036    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-serving-cert\") pod \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") "
Sep 16 00:53:56.425275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-oauth-config\") pod \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") "
Sep 16 00:53:56.425275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-config\") pod \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") "
Sep 16 00:53:56.425275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-service-ca\") pod \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") "
Sep 16 00:53:56.425275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-trusted-ca-bundle\") pod \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") "
Sep 16 00:53:56.425275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-oauth-serving-cert\") pod \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") "
Sep 16 00:53:56.425275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425259    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzrqs\" (UniqueName: \"kubernetes.io/projected/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-kube-api-access-wzrqs\") pod \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\" (UID: \"4b3b6fd3-450f-4b2e-bc08-219245bcc83c\") "
Sep 16 00:53:56.425637 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-service-ca" (OuterVolumeSpecName: "service-ca") pod "4b3b6fd3-450f-4b2e-bc08-219245bcc83c" (UID: "4b3b6fd3-450f-4b2e-bc08-219245bcc83c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:56.425637 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-config" (OuterVolumeSpecName: "console-config") pod "4b3b6fd3-450f-4b2e-bc08-219245bcc83c" (UID: "4b3b6fd3-450f-4b2e-bc08-219245bcc83c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:56.425729 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4b3b6fd3-450f-4b2e-bc08-219245bcc83c" (UID: "4b3b6fd3-450f-4b2e-bc08-219245bcc83c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:56.425729 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.425691    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4b3b6fd3-450f-4b2e-bc08-219245bcc83c" (UID: "4b3b6fd3-450f-4b2e-bc08-219245bcc83c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:56.427398 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.427375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-kube-api-access-wzrqs" (OuterVolumeSpecName: "kube-api-access-wzrqs") pod "4b3b6fd3-450f-4b2e-bc08-219245bcc83c" (UID: "4b3b6fd3-450f-4b2e-bc08-219245bcc83c"). InnerVolumeSpecName "kube-api-access-wzrqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:56.427517 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.427417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4b3b6fd3-450f-4b2e-bc08-219245bcc83c" (UID: "4b3b6fd3-450f-4b2e-bc08-219245bcc83c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:56.427517 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.427436    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4b3b6fd3-450f-4b2e-bc08-219245bcc83c" (UID: "4b3b6fd3-450f-4b2e-bc08-219245bcc83c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:56.525892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.525852    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-oauth-serving-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:53:56.525892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.525882    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrqs\" (UniqueName: \"kubernetes.io/projected/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-kube-api-access-wzrqs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:53:56.525892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.525893    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-serving-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:53:56.525892 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.525901    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-oauth-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:53:56.526161 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.525910    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-console-config\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:53:56.526161 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.525919    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:53:56.526161 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.525927    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b3b6fd3-450f-4b2e-bc08-219245bcc83c-trusted-ca-bundle\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:53:56.654666 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.654576    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b3b6fd3-450f-4b2e-bc08-219245bcc83c" containerID="76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51" exitCode=0
Sep 16 00:53:56.654666 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.654623    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bdd56d6-727qs"
Sep 16 00:53:56.654666 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.654625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bdd56d6-727qs" event={"ID":"4b3b6fd3-450f-4b2e-bc08-219245bcc83c","Type":"ContainerDied","Data":"76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51"}
Sep 16 00:53:56.654893 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.654675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bdd56d6-727qs" event={"ID":"4b3b6fd3-450f-4b2e-bc08-219245bcc83c","Type":"ContainerDied","Data":"d76e4d831554e8fbf53fc373f2e720681e188a5982fe7d02dce1e52343dd1b92"}
Sep 16 00:53:56.654893 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.654702    2581 scope.go:117] "RemoveContainer" containerID="76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51"
Sep 16 00:53:56.663850 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.663828    2581 scope.go:117] "RemoveContainer" containerID="76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51"
Sep 16 00:53:56.664127 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:53:56.664107    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51\": container with ID starting with 76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51 not found: ID does not exist" containerID="76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51"
Sep 16 00:53:56.664170 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.664137    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51"} err="failed to get container status \"76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51\": rpc error: code = NotFound desc = could not find container \"76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51\": container with ID starting with 76f0368d9e613cbcc8187531fbae1b7cc6e735bf6e40efb716031b2232172d51 not found: ID does not exist"
Sep 16 00:53:56.676456 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.676428    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567bdd56d6-727qs"]
Sep 16 00:53:56.678975 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:56.678946    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-567bdd56d6-727qs"]
Sep 16 00:53:57.941927 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:53:57.941892    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b3b6fd3-450f-4b2e-bc08-219245bcc83c" path="/var/lib/kubelet/pods/4b3b6fd3-450f-4b2e-bc08-219245bcc83c/volumes"
Sep 16 00:54:06.750751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:06.750701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:07.547293 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:07.547256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:10.591787 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:10.591753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:10.720306 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:10.720273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:10.902727 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:10.902642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:12.004823 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:12.004789    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:13.154444 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:13.154396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:13.404543 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:13.404415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:15.143786 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:15.143753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:15.560772 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:15.560735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:15.952380 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:15.952305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:16.268646 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:16.268607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:16.401608 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:16.401567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:17.531850 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:17.531796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:19.582719 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:19.582682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:19.680996 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:19.680950    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:20.353038 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:20.353004    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:20.453723 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:20.453689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:20.955347 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:20.955314    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:21.055533 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:21.055501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:23.362226 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:23.362193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:23.420802 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:23.420761    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:26.568795 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.568759    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"]
Sep 16 00:54:26.569296 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.569149    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3b6fd3-450f-4b2e-bc08-219245bcc83c" containerName="console"
Sep 16 00:54:26.569296 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.569161    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3b6fd3-450f-4b2e-bc08-219245bcc83c" containerName="console"
Sep 16 00:54:26.569296 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.569239    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b3b6fd3-450f-4b2e-bc08-219245bcc83c" containerName="console"
Sep 16 00:54:26.573667 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.573634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:26.574010 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.573992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.576693 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.576662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 00:54:26.576862 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.576760    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 00:54:26.576862 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.576791    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 00:54:26.576999 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.576905    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 00:54:26.577152 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.577135    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mkv6l\""
Sep 16 00:54:26.580162 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.580140    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"]
Sep 16 00:54:26.683227 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.683189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp6c\" (UniqueName: \"kubernetes.io/projected/8532120d-2bb1-49d3-a693-c4a3190d4e39-kube-api-access-9pp6c\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.683406 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.683260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8532120d-2bb1-49d3-a693-c4a3190d4e39-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.683406 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.683301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8532120d-2bb1-49d3-a693-c4a3190d4e39-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.721535 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.721505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:26.784714 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.784677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8532120d-2bb1-49d3-a693-c4a3190d4e39-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.784884 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.784774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8532120d-2bb1-49d3-a693-c4a3190d4e39-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.784884 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.784809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp6c\" (UniqueName: \"kubernetes.io/projected/8532120d-2bb1-49d3-a693-c4a3190d4e39-kube-api-access-9pp6c\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.785404 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.785376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8532120d-2bb1-49d3-a693-c4a3190d4e39-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.787043 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.787020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8532120d-2bb1-49d3-a693-c4a3190d4e39-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.792243 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.792221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp6c\" (UniqueName: \"kubernetes.io/projected/8532120d-2bb1-49d3-a693-c4a3190d4e39-kube-api-access-9pp6c\") pod \"kuadrant-console-plugin-fbdd4f445-pz6vc\" (UID: \"8532120d-2bb1-49d3-a693-c4a3190d4e39\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:26.885789 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:26.885685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"
Sep 16 00:54:27.152869 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:27.152845    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc"]
Sep 16 00:54:27.154891 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:54:27.154863    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8532120d_2bb1_49d3_a693_c4a3190d4e39.slice/crio-415ec24357f13a37b9871a22ae3baa411442396ef2d032bf2aee64e39480d105 WatchSource:0}: Error finding container 415ec24357f13a37b9871a22ae3baa411442396ef2d032bf2aee64e39480d105: Status 404 returned error can't find the container with id 415ec24357f13a37b9871a22ae3baa411442396ef2d032bf2aee64e39480d105
Sep 16 00:54:27.156370 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:27.156348    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:54:27.587312 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:27.587276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:27.689858 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:27.689827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:27.812435 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:27.812400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc" event={"ID":"8532120d-2bb1-49d3-a693-c4a3190d4e39","Type":"ContainerStarted","Data":"415ec24357f13a37b9871a22ae3baa411442396ef2d032bf2aee64e39480d105"}
Sep 16 00:54:32.297407 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:32.297366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:32.398724 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:32.398689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:32.467322 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:32.467243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:32.844450 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:32.844408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc" event={"ID":"8532120d-2bb1-49d3-a693-c4a3190d4e39","Type":"ContainerStarted","Data":"2b8aaddf090f113b023c0a1c3e6226efed56a621eded5b537cc829f848db64ee"}
Sep 16 00:54:32.858466 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:32.858384    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pz6vc" podStartSLOduration=1.832350784 podStartE2EDuration="6.858363962s" totalImagesPullingTime="5.026013178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:54:32.858169624 +0000 UTC m=+691.401306601" watchObservedRunningTime="2026-09-16 00:54:32.858363962 +0000 UTC m=+691.401500938"
Sep 16 00:54:40.170730 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:40.170687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:40.268587 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:40.268546    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:40.778702 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:40.778664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:40.878560 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:40.878526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:43.160595 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:43.160561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:43.261332 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:43.261297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:48.362827 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:48.362789    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:48.461827 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:48.461789    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:52.168711 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:52.168673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:52.263394 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:52.263361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:54:52.372237 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:54:52.372191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:03.304442 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:03.304389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:03.383710 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:03.383670    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:05.971499 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:05.971458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:06.071559 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:06.071524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:20.588382 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:20.588343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:20.684215 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:20.684169    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:20.783350 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:20.783310    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:20.989376 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:20.989294    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:55:21.012679 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.012644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:55:21.012822 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.012769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.015213 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.015191    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 00:55:21.016019 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.016000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:55:21.058967 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.058934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5008c036-32a3-413a-b759-6a7f3df357f0-config-file\") pod \"limitador-limitador-84994f7ffc-knfs4\" (UID: \"5008c036-32a3-413a-b759-6a7f3df357f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.059138 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.059029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzfs\" (UniqueName: \"kubernetes.io/projected/5008c036-32a3-413a-b759-6a7f3df357f0-kube-api-access-xkzfs\") pod \"limitador-limitador-84994f7ffc-knfs4\" (UID: \"5008c036-32a3-413a-b759-6a7f3df357f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.160128 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.160093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzfs\" (UniqueName: \"kubernetes.io/projected/5008c036-32a3-413a-b759-6a7f3df357f0-kube-api-access-xkzfs\") pod \"limitador-limitador-84994f7ffc-knfs4\" (UID: \"5008c036-32a3-413a-b759-6a7f3df357f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.160128 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.160134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5008c036-32a3-413a-b759-6a7f3df357f0-config-file\") pod \"limitador-limitador-84994f7ffc-knfs4\" (UID: \"5008c036-32a3-413a-b759-6a7f3df357f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.160772 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.160753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5008c036-32a3-413a-b759-6a7f3df357f0-config-file\") pod \"limitador-limitador-84994f7ffc-knfs4\" (UID: \"5008c036-32a3-413a-b759-6a7f3df357f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.170731 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.170700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzfs\" (UniqueName: \"kubernetes.io/projected/5008c036-32a3-413a-b759-6a7f3df357f0-kube-api-access-xkzfs\") pod \"limitador-limitador-84994f7ffc-knfs4\" (UID: \"5008c036-32a3-413a-b759-6a7f3df357f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.323801 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.323757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:21.420054 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.420011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:21.517339 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.517306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:21.521046 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.521020    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-6gwh6"]
Sep 16 00:55:21.524787 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.524769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6gwh6"
Sep 16 00:55:21.527072 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.527053    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rqnv6\""
Sep 16 00:55:21.532324 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.532296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6gwh6"]
Sep 16 00:55:21.564476 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.564441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh2hq\" (UniqueName: \"kubernetes.io/projected/40f9e5ef-3991-431c-929d-01c91d27d799-kube-api-access-bh2hq\") pod \"authorino-59c974bd48-6gwh6\" (UID: \"40f9e5ef-3991-431c-929d-01c91d27d799\") " pod="kuadrant-system/authorino-59c974bd48-6gwh6"
Sep 16 00:55:21.593126 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.593094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:55:21.595662 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:55:21.595635    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5008c036_32a3_413a_b759_6a7f3df357f0.slice/crio-36a4e1cb86bbc24f6758403e0782359702eb9ca38dbf1af1259692df27a07a04 WatchSource:0}: Error finding container 36a4e1cb86bbc24f6758403e0782359702eb9ca38dbf1af1259692df27a07a04: Status 404 returned error can't find the container with id 36a4e1cb86bbc24f6758403e0782359702eb9ca38dbf1af1259692df27a07a04
Sep 16 00:55:21.665618 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.665582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh2hq\" (UniqueName: \"kubernetes.io/projected/40f9e5ef-3991-431c-929d-01c91d27d799-kube-api-access-bh2hq\") pod \"authorino-59c974bd48-6gwh6\" (UID: \"40f9e5ef-3991-431c-929d-01c91d27d799\") " pod="kuadrant-system/authorino-59c974bd48-6gwh6"
Sep 16 00:55:21.673594 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.673569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh2hq\" (UniqueName: \"kubernetes.io/projected/40f9e5ef-3991-431c-929d-01c91d27d799-kube-api-access-bh2hq\") pod \"authorino-59c974bd48-6gwh6\" (UID: \"40f9e5ef-3991-431c-929d-01c91d27d799\") " pod="kuadrant-system/authorino-59c974bd48-6gwh6"
Sep 16 00:55:21.835588 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:21.835547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6gwh6"
Sep 16 00:55:22.087803 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:22.087766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4" event={"ID":"5008c036-32a3-413a-b759-6a7f3df357f0","Type":"ContainerStarted","Data":"36a4e1cb86bbc24f6758403e0782359702eb9ca38dbf1af1259692df27a07a04"}
Sep 16 00:55:22.103314 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:22.103288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6gwh6"]
Sep 16 00:55:22.105125 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:55:22.105094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40f9e5ef_3991_431c_929d_01c91d27d799.slice/crio-a22255e6b0ce8b13bfd88ef24b8ae2f0fad8068cbeee7c34107ff03776b0543d WatchSource:0}: Error finding container a22255e6b0ce8b13bfd88ef24b8ae2f0fad8068cbeee7c34107ff03776b0543d: Status 404 returned error can't find the container with id a22255e6b0ce8b13bfd88ef24b8ae2f0fad8068cbeee7c34107ff03776b0543d
Sep 16 00:55:23.095451 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:23.095402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6gwh6" event={"ID":"40f9e5ef-3991-431c-929d-01c91d27d799","Type":"ContainerStarted","Data":"a22255e6b0ce8b13bfd88ef24b8ae2f0fad8068cbeee7c34107ff03776b0543d"}
Sep 16 00:55:24.102138 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:24.102084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4" event={"ID":"5008c036-32a3-413a-b759-6a7f3df357f0","Type":"ContainerStarted","Data":"792582a0a998a9c06d5680356e6565a160a281f52f6f4ad9fcbf012e0bb8dfd9"}
Sep 16 00:55:24.102568 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:24.102157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:24.118507 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:24.118441    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4" podStartSLOduration=1.959694436 podStartE2EDuration="4.118425633s" totalImagesPullingTime="2.158731197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:55:24.116330671 +0000 UTC m=+742.659467646" watchObservedRunningTime="2026-09-16 00:55:24.118425633 +0000 UTC m=+742.661562611"
Sep 16 00:55:28.234466 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:28.234431    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:28.289339 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:28.289309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:35.108758 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:35.108722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-knfs4"
Sep 16 00:55:40.212152 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:40.212101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:40.309622 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:40.309587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:40.684357 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:40.684313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:43.205955 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:43.205915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6gwh6" event={"ID":"40f9e5ef-3991-431c-929d-01c91d27d799","Type":"ContainerStarted","Data":"e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e"}
Sep 16 00:55:43.219713 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:43.219653    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-6gwh6" podStartSLOduration=1.8615666210000001 podStartE2EDuration="22.219636372s" totalImagesPullingTime="20.358069751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:55:43.218235782 +0000 UTC m=+761.761372757" watchObservedRunningTime="2026-09-16 00:55:43.219636372 +0000 UTC m=+761.762773377"
Sep 16 00:55:53.063090 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:53.063047    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:53.238780 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:53.238744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:54.852008 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:54.851972    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6gwh6"]
Sep 16 00:55:54.852521 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:54.852142    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-6gwh6" podUID="40f9e5ef-3991-431c-929d-01c91d27d799" containerName="authorino" containerID="cri-o://e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e" gracePeriod=30
Sep 16 00:55:55.104140 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.104076    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6gwh6"
Sep 16 00:55:55.175847 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.175813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh2hq\" (UniqueName: \"kubernetes.io/projected/40f9e5ef-3991-431c-929d-01c91d27d799-kube-api-access-bh2hq\") pod \"40f9e5ef-3991-431c-929d-01c91d27d799\" (UID: \"40f9e5ef-3991-431c-929d-01c91d27d799\") "
Sep 16 00:55:55.177961 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.177934    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f9e5ef-3991-431c-929d-01c91d27d799-kube-api-access-bh2hq" (OuterVolumeSpecName: "kube-api-access-bh2hq") pod "40f9e5ef-3991-431c-929d-01c91d27d799" (UID: "40f9e5ef-3991-431c-929d-01c91d27d799"). InnerVolumeSpecName "kube-api-access-bh2hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:55:55.267365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.267326    2581 generic.go:356] "Generic (PLEG): container finished" podID="40f9e5ef-3991-431c-929d-01c91d27d799" containerID="e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e" exitCode=0
Sep 16 00:55:55.267365 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.267373    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6gwh6"
Sep 16 00:55:55.267593 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.267373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6gwh6" event={"ID":"40f9e5ef-3991-431c-929d-01c91d27d799","Type":"ContainerDied","Data":"e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e"}
Sep 16 00:55:55.267593 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.267414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6gwh6" event={"ID":"40f9e5ef-3991-431c-929d-01c91d27d799","Type":"ContainerDied","Data":"a22255e6b0ce8b13bfd88ef24b8ae2f0fad8068cbeee7c34107ff03776b0543d"}
Sep 16 00:55:55.267593 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.267439    2581 scope.go:117] "RemoveContainer" containerID="e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e"
Sep 16 00:55:55.276258 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.276239    2581 scope.go:117] "RemoveContainer" containerID="e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e"
Sep 16 00:55:55.276566 ip-10-0-130-4 kubenswrapper[2581]: E0916 00:55:55.276537    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e\": container with ID starting with e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e not found: ID does not exist" containerID="e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e"
Sep 16 00:55:55.276647 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.276582    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e"} err="failed to get container status \"e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e\": rpc error: code = NotFound desc = could not find container \"e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e\": container with ID starting with e7a87b776273833a6e09d083b91f8ed0e3ed827ca220880275e4ce54eb726d8e not found: ID does not exist"
Sep 16 00:55:55.276647 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.276547    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh2hq\" (UniqueName: \"kubernetes.io/projected/40f9e5ef-3991-431c-929d-01c91d27d799-kube-api-access-bh2hq\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 00:55:55.287232 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.287203    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6gwh6"]
Sep 16 00:55:55.290022 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.289996    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6gwh6"]
Sep 16 00:55:55.941428 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:55.941391    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f9e5ef-3991-431c-929d-01c91d27d799" path="/var/lib/kubelet/pods/40f9e5ef-3991-431c-929d-01c91d27d799/volumes"
Sep 16 00:55:59.597695 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:59.597650    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:59.696103 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:59.696058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:59.773447 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:59.773412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:55:59.833469 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:55:59.833436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:02.793606 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:02.793566    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:02.891703 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:02.891664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:17.972401 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:17.972348    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:18.026911 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:18.026873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:34.417251 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:34.417151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:34.516408 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:34.516370    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:36.118422 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:36.118388    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:36.218281 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:36.218240    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:36.317926 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:36.317837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:36.506102 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:36.505991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:36.681370 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:36.681318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:36.782893 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:36.782855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:36.840919 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:36.840887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:56:47.999004 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:47.998960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:48.096220 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:48.096179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:56:59.122677 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:56:59.122631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:08.508204 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:08.508161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:09.084185 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:09.084136    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:57:14.828681 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:14.828621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:14.879311 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:14.879274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:16.606074 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:16.606040    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:17.274762 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:17.274717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:57:28.886376 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.886340    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"]
Sep 16 00:57:28.886915 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.886895    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f9e5ef-3991-431c-929d-01c91d27d799" containerName="authorino"
Sep 16 00:57:28.887000 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.886919    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f9e5ef-3991-431c-929d-01c91d27d799" containerName="authorino"
Sep 16 00:57:28.887076 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.887062    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f9e5ef-3991-431c-929d-01c91d27d799" containerName="authorino"
Sep 16 00:57:28.890035 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.890015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:28.892247 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.892224    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 00:57:28.893060 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.893043    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 00:57:28.893151 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.893062    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-spxkl\""
Sep 16 00:57:28.893200 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.893149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 00:57:28.897813 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.897778    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"]
Sep 16 00:57:28.905787 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.905761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:28.905941 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.905833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:28.905941 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.905878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:28.905941 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.905903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddae6e-11d8-426e-9a1c-8091911ca404-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:28.906120 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.905979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:28.906120 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.906014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6l54\" (UniqueName: \"kubernetes.io/projected/39ddae6e-11d8-426e-9a1c-8091911ca404-kube-api-access-s6l54\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:28.918913 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:28.918877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:29.006429 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.006429 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6l54\" (UniqueName: \"kubernetes.io/projected/39ddae6e-11d8-426e-9a1c-8091911ca404-kube-api-access-s6l54\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.006720 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.006720 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.006720 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.006720 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddae6e-11d8-426e-9a1c-8091911ca404-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.006939 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.006997 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.006940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.007033 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.007011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.008906 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.008871    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddae6e-11d8-426e-9a1c-8091911ca404-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.009034 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.009016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddae6e-11d8-426e-9a1c-8091911ca404-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.014056 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.014033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6l54\" (UniqueName: \"kubernetes.io/projected/39ddae6e-11d8-426e-9a1c-8091911ca404-kube-api-access-s6l54\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw\" (UID: \"39ddae6e-11d8-426e-9a1c-8091911ca404\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.200905 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.200810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:29.471301 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.471273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"]
Sep 16 00:57:29.473631 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:57:29.473602    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ddae6e_11d8_426e_9a1c_8091911ca404.slice/crio-56cb0bc7c5745f57e23346fd9e55a6339ae6c15658dd9e480c23fae2dbcc6e74 WatchSource:0}: Error finding container 56cb0bc7c5745f57e23346fd9e55a6339ae6c15658dd9e480c23fae2dbcc6e74: Status 404 returned error can't find the container with id 56cb0bc7c5745f57e23346fd9e55a6339ae6c15658dd9e480c23fae2dbcc6e74
Sep 16 00:57:29.736921 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.736795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw" event={"ID":"39ddae6e-11d8-426e-9a1c-8091911ca404","Type":"ContainerStarted","Data":"56cb0bc7c5745f57e23346fd9e55a6339ae6c15658dd9e480c23fae2dbcc6e74"}
Sep 16 00:57:29.779642 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:29.779596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:57:30.743119 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:30.743076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw" event={"ID":"39ddae6e-11d8-426e-9a1c-8091911ca404","Type":"ContainerStarted","Data":"5c22e4c3cdfe3ad931ef7d72f640540b544d1e7396855229c4f5241cf9d37d24"}
Sep 16 00:57:30.758966 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:30.758845    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw" podStartSLOduration=1.726365567 podStartE2EDuration="2.758830918s" totalImagesPullingTime="1.032465351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:30.758323282 +0000 UTC m=+869.301460257" watchObservedRunningTime="2026-09-16 00:57:30.758830918 +0000 UTC m=+869.301967892"
Sep 16 00:57:33.001915 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:33.001880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:34.385805 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:34.385769    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:57:39.201280 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:39.201244    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:39.201751 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:39.201305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:39.214229 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:39.214193    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:39.799364 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:39.799337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rxqgw"
Sep 16 00:57:49.807982 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:49.807945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:57:51.485457 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:51.485423    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:57:56.385351 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:57:56.385266    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:58:01.905745 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:58:01.905706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:58:01.906812 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:58:01.906789    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 00:58:07.568471 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:58:07.568434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:58:16.018050 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:58:16.018015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:58:18.183562 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:58:18.183521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:58:39.413359 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:58:39.413316    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:58:41.490455 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:58:41.490416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 00:59:38.368440 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.368357    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84cf54b799-x42l7"]
Sep 16 00:59:38.371937 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.371914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.375227 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.375205    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 00:59:38.375348 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.375255    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rqnv6\""
Sep 16 00:59:38.381739 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.381716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84cf54b799-x42l7"]
Sep 16 00:59:38.462784 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.462731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e0c0b69-1026-44a9-b250-c54c6ed9799c-tls-cert\") pod \"authorino-84cf54b799-x42l7\" (UID: \"1e0c0b69-1026-44a9-b250-c54c6ed9799c\") " pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.462980 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.462802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55n5\" (UniqueName: \"kubernetes.io/projected/1e0c0b69-1026-44a9-b250-c54c6ed9799c-kube-api-access-v55n5\") pod \"authorino-84cf54b799-x42l7\" (UID: \"1e0c0b69-1026-44a9-b250-c54c6ed9799c\") " pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.563659 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.563620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55n5\" (UniqueName: \"kubernetes.io/projected/1e0c0b69-1026-44a9-b250-c54c6ed9799c-kube-api-access-v55n5\") pod \"authorino-84cf54b799-x42l7\" (UID: \"1e0c0b69-1026-44a9-b250-c54c6ed9799c\") " pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.563857 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.563721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e0c0b69-1026-44a9-b250-c54c6ed9799c-tls-cert\") pod \"authorino-84cf54b799-x42l7\" (UID: \"1e0c0b69-1026-44a9-b250-c54c6ed9799c\") " pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.566311 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.566283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e0c0b69-1026-44a9-b250-c54c6ed9799c-tls-cert\") pod \"authorino-84cf54b799-x42l7\" (UID: \"1e0c0b69-1026-44a9-b250-c54c6ed9799c\") " pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.571004 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.570970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55n5\" (UniqueName: \"kubernetes.io/projected/1e0c0b69-1026-44a9-b250-c54c6ed9799c-kube-api-access-v55n5\") pod \"authorino-84cf54b799-x42l7\" (UID: \"1e0c0b69-1026-44a9-b250-c54c6ed9799c\") " pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.681573 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.681467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cf54b799-x42l7"
Sep 16 00:59:38.950108 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.950080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84cf54b799-x42l7"]
Sep 16 00:59:38.951622 ip-10-0-130-4 kubenswrapper[2581]: W0916 00:59:38.951593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e0c0b69_1026_44a9_b250_c54c6ed9799c.slice/crio-f1923113fd9efdff6f12b2178f69545d88616d43d4fbb8922119f69bdc020955 WatchSource:0}: Error finding container f1923113fd9efdff6f12b2178f69545d88616d43d4fbb8922119f69bdc020955: Status 404 returned error can't find the container with id f1923113fd9efdff6f12b2178f69545d88616d43d4fbb8922119f69bdc020955
Sep 16 00:59:38.953012 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:38.952989    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:59:39.370275 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:39.370225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cf54b799-x42l7" event={"ID":"1e0c0b69-1026-44a9-b250-c54c6ed9799c","Type":"ContainerStarted","Data":"f1923113fd9efdff6f12b2178f69545d88616d43d4fbb8922119f69bdc020955"}
Sep 16 00:59:40.378126 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:40.378084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cf54b799-x42l7" event={"ID":"1e0c0b69-1026-44a9-b250-c54c6ed9799c","Type":"ContainerStarted","Data":"7bd11c21095ce895537d7e27f8ba0a6ab4879e088b48d6a7d7dc8a0c4dcee37a"}
Sep 16 00:59:40.391860 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:40.391525    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84cf54b799-x42l7" podStartSLOduration=1.80114742 podStartE2EDuration="2.391504823s" totalImagesPullingTime="590.357403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:59:40.391437362 +0000 UTC m=+998.934574338" watchObservedRunningTime="2026-09-16 00:59:40.391504823 +0000 UTC m=+998.934641792"
Sep 16 00:59:40.792554 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:40.792514    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:59:41.050928 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:41.050839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:59:41.141541 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:41.141502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:59:45.179104 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:45.179065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:59:58.269516 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:58.269459    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 00:59:58.365463 ip-10-0-130-4 kubenswrapper[2581]: I0916 00:59:58.365371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:23.415715 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:23.415673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:23.775501 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:23.775446    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:25.631525 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:25.631493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:26.386076 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:26.386039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:26.486467 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:26.486432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:41.779299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:41.779259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:43.787188 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:43.787153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:43.888245 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:43.888204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:45.311395 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:45.311363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:45.420163 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:45.420111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:45.520889 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:45.520846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:45.616717 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:45.616630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:46.490589 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:46.490547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:46.594194 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:46.594147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:46.825236 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:46.825195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:47.606174 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.606138    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"]
Sep 16 01:00:47.610153 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.610131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.612415 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.612389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-2534e3-serving-cert\""
Sep 16 01:00:47.612553 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.612438    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:00:47.613371 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.613309    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7n79q\""
Sep 16 01:00:47.613371 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.613366    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 01:00:47.613576 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.613379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 01:00:47.616741 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.616719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"]
Sep 16 01:00:47.679267 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.679227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-metrics-certs\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.679451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.679271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.679451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.679334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fcd223-dd24-4ecf-992c-70b1df7b335e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.679451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.679400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvt6\" (UniqueName: \"kubernetes.io/projected/64fcd223-dd24-4ecf-992c-70b1df7b335e-kube-api-access-dwvt6\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.780855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.780817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-metrics-certs\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.780855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.780860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.781070 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.780880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fcd223-dd24-4ecf-992c-70b1df7b335e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.781070 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.780926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvt6\" (UniqueName: \"kubernetes.io/projected/64fcd223-dd24-4ecf-992c-70b1df7b335e-kube-api-access-dwvt6\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.781675 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.781651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fcd223-dd24-4ecf-992c-70b1df7b335e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.783384 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.783356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.783563 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.783547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-metrics-certs\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.788404 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.788380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvt6\" (UniqueName: \"kubernetes.io/projected/64fcd223-dd24-4ecf-992c-70b1df7b335e-kube-api-access-dwvt6\") pod \"maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:47.924623 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:47.924533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:48.197915 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:48.197787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"]
Sep 16 01:00:48.200381 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:00:48.200353    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fcd223_dd24_4ecf_992c_70b1df7b335e.slice/crio-a486b1376ea9d7ffce381e1a38b1fd853aaff88fdc6bd524bd53885c2313f651 WatchSource:0}: Error finding container a486b1376ea9d7ffce381e1a38b1fd853aaff88fdc6bd524bd53885c2313f651: Status 404 returned error can't find the container with id a486b1376ea9d7ffce381e1a38b1fd853aaff88fdc6bd524bd53885c2313f651
Sep 16 01:00:48.733901 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:48.733862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s" event={"ID":"64fcd223-dd24-4ecf-992c-70b1df7b335e","Type":"ContainerStarted","Data":"a486b1376ea9d7ffce381e1a38b1fd853aaff88fdc6bd524bd53885c2313f651"}
Sep 16 01:00:48.961800 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:48.961767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:49.066578 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:49.066530    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:50.190352 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:50.190296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:50.290654 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:50.290620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:50.391380 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:50.391347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:51.213186 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:51.213157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:00:51.755428 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:51.755394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s" event={"ID":"64fcd223-dd24-4ecf-992c-70b1df7b335e","Type":"ContainerStarted","Data":"4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704"}
Sep 16 01:00:51.755632 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:51.755528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:00:51.774182 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:51.774105    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s" podStartSLOduration=1.6065304280000001 podStartE2EDuration="4.774083415s" totalImagesPullingTime="3.167552987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:51.772615594 +0000 UTC m=+1070.315752571" watchObservedRunningTime="2026-09-16 01:00:51.774083415 +0000 UTC m=+1070.317220391"
Sep 16 01:00:51.883640 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:51.883603    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:00:57.766432 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:00:57.766355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:01:01.359931 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:01.359889    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:01.460880 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:01.460838    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:05.461304 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:05.461267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:05.569970 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:05.569933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:05.661299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:05.661262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:05.861876 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:05.861844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:08.961028 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:08.961000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:09.058470 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:09.058433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:09.658095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:09.658062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:10.320622 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:10.320581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:10.423449 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:10.423414    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:10.914918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:10.914883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:11.020772 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:11.020741    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:11.233103 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:11.233012    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:12.216166 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:12.216131    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:12.316078 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:12.316046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:14.082722 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:14.082690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:15.451689 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:15.451648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:15.795723 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:15.795691    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:16.078402 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.078323    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"]
Sep 16 01:01:16.082098 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.082076    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.084742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.084715    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 01:01:16.084863 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.084720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6d0d72-openshift-default-dockercfg-gzfzd\""
Sep 16 01:01:16.095049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.095014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"]
Sep 16 01:01:16.104245 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.104214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:16.252636 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.252599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-socket\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.252858 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.252664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-data\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.252858 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.252719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-podinfo\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.252858 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.252816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-kube-api-access-8w8tx\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.253048 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.252867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-credential-socket\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.253048 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.252906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-certs\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.253048 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.252977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-envoy\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.253048 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.253033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-token\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.253230 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.253059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f88175-8d05-4b62-9a99-56116a0549b3-istiod-ca-cert\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.353950 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.353862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-socket\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.353950 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.353908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-data\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354165 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-podinfo\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354165 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-kube-api-access-8w8tx\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354165 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-credential-socket\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354337 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-certs\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354337 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-envoy\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354337 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-data\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354531 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-socket\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354531 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-token\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354531 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f88175-8d05-4b62-9a99-56116a0549b3-istiod-ca-cert\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-credential-socket\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.354701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.354672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-certs\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.355081 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.355062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f88175-8d05-4b62-9a99-56116a0549b3-istiod-ca-cert\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.356804 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.356784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-podinfo\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.356889 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.356814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-envoy\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.361891 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.361865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-token\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.362035 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.362018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-kube-api-access-8w8tx\") pod \"e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") " pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.394029 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.393987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:16.606823 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.606732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:16.670769 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.670742    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"]
Sep 16 01:01:16.672958 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:01:16.672927    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f88175_8d05_4b62_9a99_56116a0549b3.slice/crio-5e6707c377b0f485008a63f44fa6fbf4b7c1a29e93ec1fdff393ae145e7156ba WatchSource:0}: Error finding container 5e6707c377b0f485008a63f44fa6fbf4b7c1a29e93ec1fdff393ae145e7156ba: Status 404 returned error can't find the container with id 5e6707c377b0f485008a63f44fa6fbf4b7c1a29e93ec1fdff393ae145e7156ba
Sep 16 01:01:16.881898 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:16.881811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" event={"ID":"f8f88175-8d05-4b62-9a99-56116a0549b3","Type":"ContainerStarted","Data":"5e6707c377b0f485008a63f44fa6fbf4b7c1a29e93ec1fdff393ae145e7156ba"}
Sep 16 01:01:20.575433 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:20.575390    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:01:20.575835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:20.575497    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:01:20.575835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:20.575545    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:01:20.906284 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:20.906184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" event={"ID":"f8f88175-8d05-4b62-9a99-56116a0549b3","Type":"ContainerStarted","Data":"17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8"}
Sep 16 01:01:20.926635 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:20.926578    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podStartSLOduration=1.026575629 podStartE2EDuration="4.926557802s" totalImagesPullingTime="3.899982173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:20.925104316 +0000 UTC m=+1099.468241289" watchObservedRunningTime="2026-09-16 01:01:20.926557802 +0000 UTC m=+1099.469694776"
Sep 16 01:01:21.394765 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:21.394723    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:21.399541 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:21.399513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:21.912210 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:21.912178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:21.913102 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:21.913083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:22.724097 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:22.724059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:23.954057 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:23.954017    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:24.053544 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.053505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:24.289084 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.289043    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"]
Sep 16 01:01:24.311960 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.311921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"]
Sep 16 01:01:24.312127 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.312045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.314372 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.314349    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6d0d72-serving-cert\""
Sep 16 01:01:24.323455 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.323431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-metrics-certs\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.323595 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.323462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8qc\" (UniqueName: \"kubernetes.io/projected/ddc43567-c96f-4200-870a-007ab24d462e-kube-api-access-5p8qc\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.323595 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.323557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.323678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.323651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc43567-c96f-4200-870a-007ab24d462e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.376654 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.376585    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:24.424644 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.424609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc43567-c96f-4200-870a-007ab24d462e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.424846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.424654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-metrics-certs\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.424846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.424671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8qc\" (UniqueName: \"kubernetes.io/projected/ddc43567-c96f-4200-870a-007ab24d462e-kube-api-access-5p8qc\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.424846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.424713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.424846 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:01:24.424823    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-serving-cert: secret "maas-api-e2e-inf-b-6d0d72-serving-cert" not found
Sep 16 01:01:24.425063 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:01:24.424891    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls podName:ddc43567-c96f-4200-870a-007ab24d462e nodeName:}" failed. No retries permitted until 2026-09-16 01:01:24.924873466 +0000 UTC m=+1103.468010422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls") pod "maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp" (UID: "ddc43567-c96f-4200-870a-007ab24d462e") : secret "maas-api-e2e-inf-b-6d0d72-serving-cert" not found
Sep 16 01:01:24.425438 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.425415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc43567-c96f-4200-870a-007ab24d462e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.427258 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.427239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-metrics-certs\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.435117 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.435094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8qc\" (UniqueName: \"kubernetes.io/projected/ddc43567-c96f-4200-870a-007ab24d462e-kube-api-access-5p8qc\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.477530 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.477491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:24.607946 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.607864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:24.828870 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.828838    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:24.923299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.923205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:24.929333 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.929308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:24.931919 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:24.931897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:25.222805 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:25.222713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:25.311658 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:25.311628    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:25.491342 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:25.491315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"]
Sep 16 01:01:25.492950 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:01:25.492912    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc43567_c96f_4200_870a_007ab24d462e.slice/crio-0869739d350a2db974e19ae88e9eb1f82ee7381e469e88c29daf7f9e63ccb34e WatchSource:0}: Error finding container 0869739d350a2db974e19ae88e9eb1f82ee7381e469e88c29daf7f9e63ccb34e: Status 404 returned error can't find the container with id 0869739d350a2db974e19ae88e9eb1f82ee7381e469e88c29daf7f9e63ccb34e
Sep 16 01:01:25.940532 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:25.940499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp" event={"ID":"ddc43567-c96f-4200-870a-007ab24d462e","Type":"ContainerStarted","Data":"0869739d350a2db974e19ae88e9eb1f82ee7381e469e88c29daf7f9e63ccb34e"}
Sep 16 01:01:26.116393 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:26.116360    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:26.943143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:26.943108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp" event={"ID":"ddc43567-c96f-4200-870a-007ab24d462e","Type":"ContainerStarted","Data":"d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab"}
Sep 16 01:01:26.943614 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:26.943321    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:26.960819 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:26.960766    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp" podStartSLOduration=2.562313111 podStartE2EDuration="2.960749749s" totalImagesPullingTime="398.436638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:26.96006803 +0000 UTC m=+1105.503205043" watchObservedRunningTime="2026-09-16 01:01:26.960749749 +0000 UTC m=+1105.503886723"
Sep 16 01:01:27.439758 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:27.439720    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:29.384725 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.384683    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"]
Sep 16 01:01:29.387881 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.387855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.390064 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.390038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"default-dockercfg-67hvz\""
Sep 16 01:01:29.390931 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.390907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"openshift-service-ca.crt\""
Sep 16 01:01:29.391049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.390967    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"simulator-w0-96c840-kserve-self-signed-certs\""
Sep 16 01:01:29.391049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.391030    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"kube-root-ca.crt\""
Sep 16 01:01:29.396821 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.396705    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"]
Sep 16 01:01:29.417703 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.417670    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:29.477235 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.477198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-home\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.477417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.477249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-tmp-dir\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.477417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.477361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-model-cache\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.477417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.477395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b50e2f6-477d-4780-8242-b8a13f011411-tls-certs\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.477550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.477433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-dshm\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.477550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.477513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdzsq\" (UniqueName: \"kubernetes.io/projected/5b50e2f6-477d-4780-8242-b8a13f011411-kube-api-access-gdzsq\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.578704 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.578665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-tmp-dir\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.578969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.578721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-model-cache\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.578969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.578766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b50e2f6-477d-4780-8242-b8a13f011411-tls-certs\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.578969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.578816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-dshm\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.578969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.578870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzsq\" (UniqueName: \"kubernetes.io/projected/5b50e2f6-477d-4780-8242-b8a13f011411-kube-api-access-gdzsq\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.578969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.578913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-home\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.579252 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.579185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-tmp-dir\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.579363 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.579338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-model-cache\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.579427 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.579361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-home\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.581247 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.581221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-dshm\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.581413 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.581396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b50e2f6-477d-4780-8242-b8a13f011411-tls-certs\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.589100 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.589074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzsq\" (UniqueName: \"kubernetes.io/projected/5b50e2f6-477d-4780-8242-b8a13f011411-kube-api-access-gdzsq\") pod \"simulator-w0-96c840-kserve-7d445594f7-82z9x\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") " pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.699357 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.699259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:29.885065 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.885021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:01:29.969648 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:29.969620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"]
Sep 16 01:01:29.972144 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:01:29.972108    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b50e2f6_477d_4780_8242_b8a13f011411.slice/crio-f34867e95501e697a4e9c63f6eac9dc82c053d99d57b40bde1d817a38442b62b WatchSource:0}: Error finding container f34867e95501e697a4e9c63f6eac9dc82c053d99d57b40bde1d817a38442b62b: Status 404 returned error can't find the container with id f34867e95501e697a4e9c63f6eac9dc82c053d99d57b40bde1d817a38442b62b
Sep 16 01:01:30.975194 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:30.975159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x" event={"ID":"5b50e2f6-477d-4780-8242-b8a13f011411","Type":"ContainerStarted","Data":"a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e"}
Sep 16 01:01:30.975583 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:30.975209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x" event={"ID":"5b50e2f6-477d-4780-8242-b8a13f011411","Type":"ContainerStarted","Data":"f34867e95501e697a4e9c63f6eac9dc82c053d99d57b40bde1d817a38442b62b"}
Sep 16 01:01:30.992850 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:30.992799    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x" podStartSLOduration=1.992784827 podStartE2EDuration="1.992784827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:30.990758612 +0000 UTC m=+1109.533895587" watchObservedRunningTime="2026-09-16 01:01:30.992784827 +0000 UTC m=+1109.535921802"
Sep 16 01:01:32.953771 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:32.953740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:01:37.954643 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:37.954605    2581 patch_prober.go:28] interesting pod/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 01:01:37.955086 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:37.954696    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 01:01:38.317970 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:38.317935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:39.699457 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:39.699416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:39.699979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:39.699476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:39.712535 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:39.712501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:40.031852 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:40.031815    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerID="17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8" exitCode=0
Sep 16 01:01:40.032084 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:40.031889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" event={"ID":"f8f88175-8d05-4b62-9a99-56116a0549b3","Type":"ContainerDied","Data":"17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8"}
Sep 16 01:01:40.032403 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:40.032377    2581 scope.go:117] "RemoveContainer" containerID="17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8"
Sep 16 01:01:40.033679 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:40.033651    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:01:40.033752 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:40.033721    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:01:40.033752 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:40.033749    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:01:40.043847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:40.043824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:01:41.039418 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:41.039375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" event={"ID":"f8f88175-8d05-4b62-9a99-56116a0549b3","Type":"ContainerStarted","Data":"c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc"}
Sep 16 01:01:41.395084 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:41.394975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:41.395272 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:41.395226    2581 patch_prober.go:28] interesting pod/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 16 01:01:41.395352 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:41.395284    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 16 01:01:42.394468 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:42.394423    2581 patch_prober.go:28] interesting pod/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 16 01:01:42.394903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:42.394513    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 16 01:01:43.395171 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:43.395130    2581 patch_prober.go:28] interesting pod/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 16 01:01:43.395635 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:43.395191    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 16 01:01:44.394661 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:44.394621    2581 patch_prober.go:28] interesting pod/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 16 01:01:44.394854 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:44.394695    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 16 01:01:44.687743 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:44.687651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:01:45.395418 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:45.395379    2581 patch_prober.go:28] interesting pod/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 16 01:01:45.395638 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:45.395436    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 16 01:01:46.304024 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:46.303989    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:46.398351 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:46.398323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:46.398715 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:46.398694    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:46.399295 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:46.399274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:01:47.669216 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:47.669176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:48.472895 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:48.472858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:48.574284 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:48.574232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:01:48.823062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:01:48.823029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:02:05.988309 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:05.988272    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"]
Sep 16 01:02:05.991027 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:05.991007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:05.994319 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:05.994297    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d58169\"/\"openshift-service-ca.crt\""
Sep 16 01:02:05.995200 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:05.995183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d58169\"/\"default-dockercfg-42zfm\""
Sep 16 01:02:05.995358 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:05.995338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d58169\"/\"kube-root-ca.crt\""
Sep 16 01:02:05.995443 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:05.995334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d58169\"/\"test-model-d58169-kserve-self-signed-certs\""
Sep 16 01:02:06.004451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.004414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"]
Sep 16 01:02:06.031336 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.031296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:02:06.039299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.039268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tmp-dir\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.039434 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.039342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxxv\" (UniqueName: \"kubernetes.io/projected/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-kube-api-access-gnxxv\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.039434 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.039370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-model-cache\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.039434 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.039392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-dshm\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.039434 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.039412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tls-certs\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.039620 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.039477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-home\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.140665 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.140630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxxv\" (UniqueName: \"kubernetes.io/projected/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-kube-api-access-gnxxv\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.140665 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.140668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-model-cache\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.140882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.140791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-dshm\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.140882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.140830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tls-certs\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.140961 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.140886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-home\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.141010 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.140964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tmp-dir\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.141305 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.141281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-model-cache\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.141383 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.141328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-home\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.141383 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.141359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tmp-dir\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.143235 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.143216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-dshm\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.143401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.143385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tls-certs\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.148032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.148010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxxv\" (UniqueName: \"kubernetes.io/projected/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-kube-api-access-gnxxv\") pod \"test-model-d58169-kserve-67d8f69d4c-psk7t\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") " pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.303228 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.303189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:06.573285 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:06.573257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"]
Sep 16 01:02:06.574507 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:02:06.574458    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe6d59bf_2cd7_49e6_816e_9ff9e30c985a.slice/crio-fc591f151bb2211528d883c472bcd4533a2fb34c9c606ec48fc096050247c506 WatchSource:0}: Error finding container fc591f151bb2211528d883c472bcd4533a2fb34c9c606ec48fc096050247c506: Status 404 returned error can't find the container with id fc591f151bb2211528d883c472bcd4533a2fb34c9c606ec48fc096050247c506
Sep 16 01:02:07.184525 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:07.184474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t" event={"ID":"be6d59bf-2cd7-49e6-816e-9ff9e30c985a","Type":"ContainerStarted","Data":"5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad"}
Sep 16 01:02:07.184913 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:07.184536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t" event={"ID":"be6d59bf-2cd7-49e6-816e-9ff9e30c985a","Type":"ContainerStarted","Data":"fc591f151bb2211528d883c472bcd4533a2fb34c9c606ec48fc096050247c506"}
Sep 16 01:02:07.202954 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:07.202905    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t" podStartSLOduration=2.202890726 podStartE2EDuration="2.202890726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:02:07.200508051 +0000 UTC m=+1145.743645028" watchObservedRunningTime="2026-09-16 01:02:07.202890726 +0000 UTC m=+1145.746027695"
Sep 16 01:02:10.886310 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:10.886267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:02:16.303652 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:16.303608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:16.304141 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:16.303670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:16.316740 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:16.316712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:17.251496 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:17.251442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:02:24.378015 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:24.377929    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:02:30.080243 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:30.080205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:02:36.790657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:36.790618    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:02:49.094275 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:49.094228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:02:56.587461 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:02:56.587413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:03:01.941832 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:01.941803    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:03:01.945497 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:01.945462    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:03:18.991638 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:18.991602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:03:23.713006 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:23.712975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:03:44.886964 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:44.886919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:03:48.351986 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.351946    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9052-6d78ac"]
Sep 16 01:03:48.355564 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.355544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:48.358175 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.358146    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c6q92\""
Sep 16 01:03:48.361010 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.360984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9052-6d78ac"]
Sep 16 01:03:48.539510 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.539439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6rp\" (UniqueName: \"kubernetes.io/projected/55490512-56d3-4391-a863-e6fada8b2484-kube-api-access-2n6rp\") pod \"test-curl-9052-6d78ac\" (UID: \"55490512-56d3-4391-a863-e6fada8b2484\") " pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:48.641050 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.640959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6rp\" (UniqueName: \"kubernetes.io/projected/55490512-56d3-4391-a863-e6fada8b2484-kube-api-access-2n6rp\") pod \"test-curl-9052-6d78ac\" (UID: \"55490512-56d3-4391-a863-e6fada8b2484\") " pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:48.648819 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.648793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6rp\" (UniqueName: \"kubernetes.io/projected/55490512-56d3-4391-a863-e6fada8b2484-kube-api-access-2n6rp\") pod \"test-curl-9052-6d78ac\" (UID: \"55490512-56d3-4391-a863-e6fada8b2484\") " pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:48.666664 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.666635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:48.935231 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:48.935200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9052-6d78ac"]
Sep 16 01:03:48.937250 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:03:48.937217    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55490512_56d3_4391_a863_e6fada8b2484.slice/crio-8a7184ba81e69cb149f1df83ea388b814860c34a87239fbb5521909b65e23b8f WatchSource:0}: Error finding container 8a7184ba81e69cb149f1df83ea388b814860c34a87239fbb5521909b65e23b8f: Status 404 returned error can't find the container with id 8a7184ba81e69cb149f1df83ea388b814860c34a87239fbb5521909b65e23b8f
Sep 16 01:03:49.741180 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:49.741137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9052-6d78ac" event={"ID":"55490512-56d3-4391-a863-e6fada8b2484","Type":"ContainerStarted","Data":"8a7184ba81e69cb149f1df83ea388b814860c34a87239fbb5521909b65e23b8f"}
Sep 16 01:03:50.200933 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:50.200892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:03:51.757706 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:51.757666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9052-6d78ac" event={"ID":"55490512-56d3-4391-a863-e6fada8b2484","Type":"ContainerStarted","Data":"9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257"}
Sep 16 01:03:51.773320 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:51.773264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9052-6d78ac" podStartSLOduration=1.699143968 podStartE2EDuration="3.773248319s" totalImagesPullingTime="2.074104351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:03:51.771534922 +0000 UTC m=+1250.314671897" watchObservedRunningTime="2026-09-16 01:03:51.773248319 +0000 UTC m=+1250.316385293"
Sep 16 01:03:52.363823 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.363753    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9052-6d78ac"]
Sep 16 01:03:52.366558 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.366529    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9052-6d78ac"]
Sep 16 01:03:52.489619 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.489530    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9052-4b812d"]
Sep 16 01:03:52.490026 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.490006    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55490512-56d3-4391-a863-e6fada8b2484" containerName="test-curl-9052-6d78ac"
Sep 16 01:03:52.490118 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.490030    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55490512-56d3-4391-a863-e6fada8b2484" containerName="test-curl-9052-6d78ac"
Sep 16 01:03:52.490172 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.490135    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="55490512-56d3-4391-a863-e6fada8b2484" containerName="test-curl-9052-6d78ac"
Sep 16 01:03:52.493397 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.493379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:52.498417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.498381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9052-4b812d"]
Sep 16 01:03:52.577552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.577511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfchk\" (UniqueName: \"kubernetes.io/projected/20ce4e03-c9df-406b-b2b1-6ef494d834b7-kube-api-access-sfchk\") pod \"test-curl-9052-4b812d\" (UID: \"20ce4e03-c9df-406b-b2b1-6ef494d834b7\") " pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:52.678305 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.678267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfchk\" (UniqueName: \"kubernetes.io/projected/20ce4e03-c9df-406b-b2b1-6ef494d834b7-kube-api-access-sfchk\") pod \"test-curl-9052-4b812d\" (UID: \"20ce4e03-c9df-406b-b2b1-6ef494d834b7\") " pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:52.685960 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.685927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfchk\" (UniqueName: \"kubernetes.io/projected/20ce4e03-c9df-406b-b2b1-6ef494d834b7-kube-api-access-sfchk\") pod \"test-curl-9052-4b812d\" (UID: \"20ce4e03-c9df-406b-b2b1-6ef494d834b7\") " pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:52.766925 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:52.766895    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9052-6d78ac" container="test-curl-9052-6d78ac"
Sep 16 01:03:52.805209 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:52.805171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:53.073695 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:53.073671    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9052-4b812d"]
Sep 16 01:03:53.074976 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:03:53.074942    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20ce4e03_c9df_406b_b2b1_6ef494d834b7.slice/crio-3f66b274b90c2bc70a571ad7d8b94b78422e5cb789597d089e3ebd64d2f41fc3 WatchSource:0}: Error finding container 3f66b274b90c2bc70a571ad7d8b94b78422e5cb789597d089e3ebd64d2f41fc3: Status 404 returned error can't find the container with id 3f66b274b90c2bc70a571ad7d8b94b78422e5cb789597d089e3ebd64d2f41fc3
Sep 16 01:03:53.774286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:53.774233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9052-4b812d" event={"ID":"20ce4e03-c9df-406b-b2b1-6ef494d834b7","Type":"ContainerStarted","Data":"f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd"}
Sep 16 01:03:53.774286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:53.774289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9052-4b812d" event={"ID":"20ce4e03-c9df-406b-b2b1-6ef494d834b7","Type":"ContainerStarted","Data":"3f66b274b90c2bc70a571ad7d8b94b78422e5cb789597d089e3ebd64d2f41fc3"}
Sep 16 01:03:53.774815 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:53.774344    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9052-6d78ac" podUID="55490512-56d3-4391-a863-e6fada8b2484" containerName="test-curl-9052-6d78ac" containerID="cri-o://9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257" gracePeriod=2
Sep 16 01:03:53.777099 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:53.777074    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9052-6d78ac\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="55490512-56d3-4391-a863-e6fada8b2484" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:53.793563 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:53.793514    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9052-4b812d" podStartSLOduration=1.7934760619999999 podStartE2EDuration="1.793476062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:03:53.792590253 +0000 UTC m=+1252.335727229" watchObservedRunningTime="2026-09-16 01:03:53.793476062 +0000 UTC m=+1252.336613040"
Sep 16 01:03:53.794731 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:53.794706    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9052-6d78ac\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="55490512-56d3-4391-a863-e6fada8b2484" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:54.386536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:54.386502    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9052-4b812d"]
Sep 16 01:03:54.390953 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:54.390917    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9052-4b812d"]
Sep 16 01:03:54.779241 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:54.779210    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9052-4b812d" container="test-curl-9052-4b812d"
Sep 16 01:03:55.784949 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:55.784893    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9052-4b812d" podUID="20ce4e03-c9df-406b-b2b1-6ef494d834b7" containerName="test-curl-9052-4b812d" containerID="cri-o://f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd" gracePeriod=2
Sep 16 01:03:55.787588 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:55.787560    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9052-4b812d\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="20ce4e03-c9df-406b-b2b1-6ef494d834b7" pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:55.961263 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:55.961234    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:56.106731 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.106642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n6rp\" (UniqueName: \"kubernetes.io/projected/55490512-56d3-4391-a863-e6fada8b2484-kube-api-access-2n6rp\") pod \"55490512-56d3-4391-a863-e6fada8b2484\" (UID: \"55490512-56d3-4391-a863-e6fada8b2484\") "
Sep 16 01:03:56.108817 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.108793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55490512-56d3-4391-a863-e6fada8b2484-kube-api-access-2n6rp" (OuterVolumeSpecName: "kube-api-access-2n6rp") pod "55490512-56d3-4391-a863-e6fada8b2484" (UID: "55490512-56d3-4391-a863-e6fada8b2484"). InnerVolumeSpecName "kube-api-access-2n6rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:03:56.208064 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.208024    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6rp\" (UniqueName: \"kubernetes.io/projected/55490512-56d3-4391-a863-e6fada8b2484-kube-api-access-2n6rp\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:03:56.791494 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.791445    2581 generic.go:356] "Generic (PLEG): container finished" podID="55490512-56d3-4391-a863-e6fada8b2484" containerID="9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257" exitCode=137
Sep 16 01:03:56.791979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.791524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:56.791979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.791546    2581 scope.go:117] "RemoveContainer" containerID="9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257"
Sep 16 01:03:56.800469 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.800453    2581 scope.go:117] "RemoveContainer" containerID="9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257"
Sep 16 01:03:56.800749 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:56.800730    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257\": container with ID starting with 9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257 not found: ID does not exist" containerID="9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257"
Sep 16 01:03:56.800832 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:56.800757    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257"} err="failed to get container status \"9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257\": rpc error: code = NotFound desc = could not find container \"9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257\": container with ID starting with 9ffb80ac2171d0d103db39fed168350eb7d8d2f512998932cd8b9e4b4fe20257 not found: ID does not exist"
Sep 16 01:03:56.801469 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:56.801446    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9052-6d78ac\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="55490512-56d3-4391-a863-e6fada8b2484" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:57.935659 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:57.935619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:57.937792 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:57.937761    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9052-6d78ac\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="55490512-56d3-4391-a863-e6fada8b2484" pod="opendatahub/test-curl-9052-6d78ac"
Sep 16 01:03:57.941873 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:57.941850    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55490512-56d3-4391-a863-e6fada8b2484" path="/var/lib/kubelet/pods/55490512-56d3-4391-a863-e6fada8b2484/volumes"
Sep 16 01:03:58.020884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.020847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfchk\" (UniqueName: \"kubernetes.io/projected/20ce4e03-c9df-406b-b2b1-6ef494d834b7-kube-api-access-sfchk\") pod \"20ce4e03-c9df-406b-b2b1-6ef494d834b7\" (UID: \"20ce4e03-c9df-406b-b2b1-6ef494d834b7\") "
Sep 16 01:03:58.022982 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.022949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ce4e03-c9df-406b-b2b1-6ef494d834b7-kube-api-access-sfchk" (OuterVolumeSpecName: "kube-api-access-sfchk") pod "20ce4e03-c9df-406b-b2b1-6ef494d834b7" (UID: "20ce4e03-c9df-406b-b2b1-6ef494d834b7"). InnerVolumeSpecName "kube-api-access-sfchk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:03:58.122163 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.122127    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfchk\" (UniqueName: \"kubernetes.io/projected/20ce4e03-c9df-406b-b2b1-6ef494d834b7-kube-api-access-sfchk\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:03:58.805753 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.805719    2581 generic.go:356] "Generic (PLEG): container finished" podID="20ce4e03-c9df-406b-b2b1-6ef494d834b7" containerID="f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd" exitCode=137
Sep 16 01:03:58.805938 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.805760    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9052-4b812d"
Sep 16 01:03:58.805938 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.805774    2581 scope.go:117] "RemoveContainer" containerID="f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd"
Sep 16 01:03:58.815635 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.815430    2581 scope.go:117] "RemoveContainer" containerID="f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd"
Sep 16 01:03:58.815789 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:03:58.815770    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd\": container with ID starting with f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd not found: ID does not exist" containerID="f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd"
Sep 16 01:03:58.815848 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:58.815802    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd"} err="failed to get container status \"f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd\": rpc error: code = NotFound desc = could not find container \"f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd\": container with ID starting with f3023b79921b0b9c6b911b2a129b35ac15e974d0ad33b2d7044bcc25343beacd not found: ID does not exist"
Sep 16 01:03:59.941851 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:03:59.941817    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ce4e03-c9df-406b-b2b1-6ef494d834b7" path="/var/lib/kubelet/pods/20ce4e03-c9df-406b-b2b1-6ef494d834b7/volumes"
Sep 16 01:04:11.721265 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:11.721210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:14.792417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:14.792369    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:04:18.078813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:18.078776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:37.386047 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:37.386008    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:04:39.884328 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.884282    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"]
Sep 16 01:04:39.884970 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.884950    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ce4e03-c9df-406b-b2b1-6ef494d834b7" containerName="test-curl-9052-4b812d"
Sep 16 01:04:39.885018 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.884974    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ce4e03-c9df-406b-b2b1-6ef494d834b7" containerName="test-curl-9052-4b812d"
Sep 16 01:04:39.885100 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.885088    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ce4e03-c9df-406b-b2b1-6ef494d834b7" containerName="test-curl-9052-4b812d"
Sep 16 01:04:39.888826 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.888799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:39.892322 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.892275    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"openshift-service-ca.crt\""
Sep 16 01:04:39.892679 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.892652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"default-dockercfg-t7hw2\""
Sep 16 01:04:39.893271 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.892894    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"distinct-2-w3-2534e3-kserve-self-signed-certs\""
Sep 16 01:04:39.893374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.892954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"kube-root-ca.crt\""
Sep 16 01:04:39.895832 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.895805    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"]
Sep 16 01:04:39.909218 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.909191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:39.928548 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.928493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-home\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:39.928548 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.928553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-tmp-dir\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:39.928749 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.928606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-model-cache\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:39.928749 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.928621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnw4\" (UniqueName: \"kubernetes.io/projected/ee1251d1-7d4d-410e-8010-a10bcf15691e-kube-api-access-stnw4\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:39.928749 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.928704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1251d1-7d4d-410e-8010-a10bcf15691e-tls-certs\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:39.928749 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:39.928739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-dshm\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029350 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1251d1-7d4d-410e-8010-a10bcf15691e-tls-certs\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029350 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-dshm\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029675 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-home\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029675 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-tmp-dir\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029675 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-model-cache\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnw4\" (UniqueName: \"kubernetes.io/projected/ee1251d1-7d4d-410e-8010-a10bcf15691e-kube-api-access-stnw4\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029906 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-model-cache\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029957 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-home\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.029957 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.029948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-tmp-dir\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.031722 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.031700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-dshm\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.032052 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.032035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1251d1-7d4d-410e-8010-a10bcf15691e-tls-certs\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.036961 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.036939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnw4\" (UniqueName: \"kubernetes.io/projected/ee1251d1-7d4d-410e-8010-a10bcf15691e-kube-api-access-stnw4\") pod \"distinct-2-w3-2534e3-kserve-77b65f965-qzgkq\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.201243 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.201154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:40.472553 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:40.472529    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"]
Sep 16 01:04:40.474664 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:04:40.474629    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee1251d1_7d4d_410e_8010_a10bcf15691e.slice/crio-f22cd5f032a538a22f92b836aae2218c8fbf2c16294f4a4c41312a19d118c2f1 WatchSource:0}: Error finding container f22cd5f032a538a22f92b836aae2218c8fbf2c16294f4a4c41312a19d118c2f1: Status 404 returned error can't find the container with id f22cd5f032a538a22f92b836aae2218c8fbf2c16294f4a4c41312a19d118c2f1
Sep 16 01:04:41.035765 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:41.035731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq" event={"ID":"ee1251d1-7d4d-410e-8010-a10bcf15691e","Type":"ContainerStarted","Data":"9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d"}
Sep 16 01:04:41.036157 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:41.035772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq" event={"ID":"ee1251d1-7d4d-410e-8010-a10bcf15691e","Type":"ContainerStarted","Data":"f22cd5f032a538a22f92b836aae2218c8fbf2c16294f4a4c41312a19d118c2f1"}
Sep 16 01:04:41.051203 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:41.051146    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq" podStartSLOduration=2.051125212 podStartE2EDuration="2.051125212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:04:41.050306501 +0000 UTC m=+1299.593443476" watchObservedRunningTime="2026-09-16 01:04:41.051125212 +0000 UTC m=+1299.594262188"
Sep 16 01:04:42.562302 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:42.562268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:42.659832 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:42.659796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:43.045815 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:43.045776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:46.594706 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:46.594671    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:48.829989 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:48.829958    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:48.887330 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:48.887296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:49.073266 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.073228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:49.350291 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.350245    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"]
Sep 16 01:04:49.354179 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.354152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.358496 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.358452    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f097ba39-openshift-default-dockercfg-44vlm\""
Sep 16 01:04:49.372425 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.372396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"]
Sep 16 01:04:49.388937 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.388903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:49.419595 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-credential-socket\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.419762 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-token\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.419762 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istiod-ca-cert\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.419762 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-envoy\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.419897 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-data\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.419938 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf57m\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-kube-api-access-xf57m\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.419938 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-podinfo\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.420016 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.419967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-socket\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.420056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.420036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-certs\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.520934 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.520883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istiod-ca-cert\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521142 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.520965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-envoy\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521142 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.520994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-data\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521142 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf57m\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-kube-api-access-xf57m\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521142 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-podinfo\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521142 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-socket\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521428 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-certs\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521428 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-credential-socket\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521428 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-token\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-data\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-certs\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521729 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-credential-socket\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istiod-ca-cert\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.521985 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.521889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-socket\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.523673 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.523652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-envoy\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.523867 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.523849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-podinfo\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.528686 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.528667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-token\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.528815 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.528799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf57m\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-kube-api-access-xf57m\") pod \"e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") " pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.594098 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.594056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:49.665906 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.665811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:49.948846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.943696    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"]
Sep 16 01:04:49.951241 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:04:49.951211    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c2140a_8a33_4bf7_96b7_fe8f4b9dc469.slice/crio-3e868b2ffff528c158c3c94ee5e97081e82317b33af4332b0480f14fec0467b9 WatchSource:0}: Error finding container 3e868b2ffff528c158c3c94ee5e97081e82317b33af4332b0480f14fec0467b9: Status 404 returned error can't find the container with id 3e868b2ffff528c158c3c94ee5e97081e82317b33af4332b0480f14fec0467b9
Sep 16 01:04:49.953677 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.953641    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:04:49.953768 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.953718    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:04:49.953768 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:49.953746    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:04:50.085134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:50.085092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" event={"ID":"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469","Type":"ContainerStarted","Data":"6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866"}
Sep 16 01:04:50.085331 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:50.085148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" event={"ID":"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469","Type":"ContainerStarted","Data":"3e868b2ffff528c158c3c94ee5e97081e82317b33af4332b0480f14fec0467b9"}
Sep 16 01:04:50.105841 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:50.105770    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podStartSLOduration=1.105748411 podStartE2EDuration="1.105748411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:04:50.103600961 +0000 UTC m=+1308.646737946" watchObservedRunningTime="2026-09-16 01:04:50.105748411 +0000 UTC m=+1308.648885387"
Sep 16 01:04:50.202307 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:50.202202    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:50.202307 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:50.202275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:50.221431 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:50.221398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:50.666695 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:50.666658    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:51.016930 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:51.016898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:51.102493 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:51.102450    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:04:51.532198 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:51.532161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:51.634340 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:51.634304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:51.671156 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:51.671125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:51.913156 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:51.913065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:52.097032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:52.096994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:52.097957 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:52.097939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:04:52.723269 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:52.723232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:53.409046 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:53.409009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:54.068638 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.068602    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"]
Sep 16 01:04:54.089559 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.089514    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:54.300313 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.300281    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"]
Sep 16 01:04:54.316380 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.316352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.318270 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.318233    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"]
Sep 16 01:04:54.318981 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.318912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f097ba39-serving-cert\""
Sep 16 01:04:54.336114 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.336072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:54.371414 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.371379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82w4\" (UniqueName: \"kubernetes.io/projected/4be65ea6-31eb-4102-a87b-380ec97fc374-kube-api-access-s82w4\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.371628 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.371464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-metrics-certs\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.371675 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.371626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be65ea6-31eb-4102-a87b-380ec97fc374-openshift-service-ca\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.371675 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.371661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.434041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.434002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:54.473197 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.473162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be65ea6-31eb-4102-a87b-380ec97fc374-openshift-service-ca\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.473379 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.473202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.473379 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.473224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82w4\" (UniqueName: \"kubernetes.io/projected/4be65ea6-31eb-4102-a87b-380ec97fc374-kube-api-access-s82w4\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.473379 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.473287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-metrics-certs\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.473379 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:04:54.473332    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-serving-cert: secret "maas-api-e2e-ait-f097ba39-serving-cert" not found
Sep 16 01:04:54.473568 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:04:54.473415    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls podName:4be65ea6-31eb-4102-a87b-380ec97fc374 nodeName:}" failed. No retries permitted until 2026-09-16 01:04:54.97339336 +0000 UTC m=+1313.516530312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls") pod "maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss" (UID: "4be65ea6-31eb-4102-a87b-380ec97fc374") : secret "maas-api-e2e-ait-f097ba39-serving-cert" not found
Sep 16 01:04:54.474019 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.474002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be65ea6-31eb-4102-a87b-380ec97fc374-openshift-service-ca\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.475928 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.475910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-metrics-certs\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.481229 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.481208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82w4\" (UniqueName: \"kubernetes.io/projected/4be65ea6-31eb-4102-a87b-380ec97fc374-kube-api-access-s82w4\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.564323 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.564290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:54.779055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.779019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:54.879631 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.879598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:04:54.978834 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.978795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:54.981236 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:54.981214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls\") pod \"maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:55.230816 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:55.230734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:55.502722 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:55.502688    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"]
Sep 16 01:04:55.504577 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:04:55.504549    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be65ea6_31eb_4102_a87b_380ec97fc374.slice/crio-323b6505834f6dfb063839462ed4d1352f31ccd8edcd0c6239cc1b2e980119ad WatchSource:0}: Error finding container 323b6505834f6dfb063839462ed4d1352f31ccd8edcd0c6239cc1b2e980119ad: Status 404 returned error can't find the container with id 323b6505834f6dfb063839462ed4d1352f31ccd8edcd0c6239cc1b2e980119ad
Sep 16 01:04:55.505883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:55.505864    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 01:04:56.119316 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:56.119287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss" event={"ID":"4be65ea6-31eb-4102-a87b-380ec97fc374","Type":"ContainerStarted","Data":"323b6505834f6dfb063839462ed4d1352f31ccd8edcd0c6239cc1b2e980119ad"}
Sep 16 01:04:57.125903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:57.125863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss" event={"ID":"4be65ea6-31eb-4102-a87b-380ec97fc374","Type":"ContainerStarted","Data":"f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4"}
Sep 16 01:04:57.126318 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:57.125994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:04:57.141772 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:04:57.141714    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss" podStartSLOduration=2.5806934740000003 podStartE2EDuration="3.141694762s" totalImagesPullingTime="561.001288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:04:57.141617051 +0000 UTC m=+1315.684754050" watchObservedRunningTime="2026-09-16 01:04:57.141694762 +0000 UTC m=+1315.684831737"
Sep 16 01:05:03.137980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:03.137943    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:05:04.888388 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:04.888355    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:05:09.090412 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.090347    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t" podUID="be6d59bf-2cd7-49e6-816e-9ff9e30c985a" containerName="main" containerID="cri-o://5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad" gracePeriod=45
Sep 16 01:05:09.389268 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.389239    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:05:09.527567 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.527528    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-model-cache\") pod \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") "
Sep 16 01:05:09.527774 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.527619    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tls-certs\") pod \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") "
Sep 16 01:05:09.527774 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.527685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnxxv\" (UniqueName: \"kubernetes.io/projected/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-kube-api-access-gnxxv\") pod \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") "
Sep 16 01:05:09.527912 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.527855    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-dshm\") pod \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") "
Sep 16 01:05:09.527912 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.527895    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tmp-dir\") pod \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") "
Sep 16 01:05:09.528025 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.527849    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-model-cache" (OuterVolumeSpecName: "model-cache") pod "be6d59bf-2cd7-49e6-816e-9ff9e30c985a" (UID: "be6d59bf-2cd7-49e6-816e-9ff9e30c985a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:09.528025 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.527914    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-home\") pod \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\" (UID: \"be6d59bf-2cd7-49e6-816e-9ff9e30c985a\") "
Sep 16 01:05:09.528146 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.528079    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-home" (OuterVolumeSpecName: "home") pod "be6d59bf-2cd7-49e6-816e-9ff9e30c985a" (UID: "be6d59bf-2cd7-49e6-816e-9ff9e30c985a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:09.528146 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.528096    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be6d59bf-2cd7-49e6-816e-9ff9e30c985a" (UID: "be6d59bf-2cd7-49e6-816e-9ff9e30c985a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:09.528374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.528354    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tmp-dir\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.528432 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.528378    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-home\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.528432 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.528392    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-model-cache\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.530051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.530022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-dshm" (OuterVolumeSpecName: "dshm") pod "be6d59bf-2cd7-49e6-816e-9ff9e30c985a" (UID: "be6d59bf-2cd7-49e6-816e-9ff9e30c985a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:09.530051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.530042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be6d59bf-2cd7-49e6-816e-9ff9e30c985a" (UID: "be6d59bf-2cd7-49e6-816e-9ff9e30c985a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:09.530184 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.530072    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-kube-api-access-gnxxv" (OuterVolumeSpecName: "kube-api-access-gnxxv") pod "be6d59bf-2cd7-49e6-816e-9ff9e30c985a" (UID: "be6d59bf-2cd7-49e6-816e-9ff9e30c985a"). InnerVolumeSpecName "kube-api-access-gnxxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:09.570575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.570533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:09.629954 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.629853    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-tls-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.629954 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.629891    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxxv\" (UniqueName: \"kubernetes.io/projected/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-kube-api-access-gnxxv\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.629954 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.629905    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d59bf-2cd7-49e6-816e-9ff9e30c985a-dshm\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.661099 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:09.661059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:10.207151 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.207053    2581 generic.go:356] "Generic (PLEG): container finished" podID="be6d59bf-2cd7-49e6-816e-9ff9e30c985a" containerID="5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad" exitCode=0
Sep 16 01:05:10.207151 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.207101    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"
Sep 16 01:05:10.207763 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.207140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t" event={"ID":"be6d59bf-2cd7-49e6-816e-9ff9e30c985a","Type":"ContainerDied","Data":"5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad"}
Sep 16 01:05:10.207763 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.207189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t" event={"ID":"be6d59bf-2cd7-49e6-816e-9ff9e30c985a","Type":"ContainerDied","Data":"fc591f151bb2211528d883c472bcd4533a2fb34c9c606ec48fc096050247c506"}
Sep 16 01:05:10.207763 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.207211    2581 scope.go:117] "RemoveContainer" containerID="5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad"
Sep 16 01:05:10.216737 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.216716    2581 scope.go:117] "RemoveContainer" containerID="5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad"
Sep 16 01:05:10.217082 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:05:10.217061    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad\": container with ID starting with 5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad not found: ID does not exist" containerID="5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad"
Sep 16 01:05:10.217165 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.217091    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad"} err="failed to get container status \"5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad\": rpc error: code = NotFound desc = could not find container \"5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad\": container with ID starting with 5f4be70672d2eda2722b3f7196f74fa65e386b7a46a686460d770b1ef34641ad not found: ID does not exist"
Sep 16 01:05:10.223258 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.223232    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"]
Sep 16 01:05:10.228816 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:10.228792    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d58169/test-model-d58169-kserve-67d8f69d4c-psk7t"]
Sep 16 01:05:11.948679 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:11.948644    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be6d59bf-2cd7-49e6-816e-9ff9e30c985a" path="/var/lib/kubelet/pods/be6d59bf-2cd7-49e6-816e-9ff9e30c985a/volumes"
Sep 16 01:05:16.245439 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:16.245406    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerID="6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866" exitCode=0
Sep 16 01:05:16.245878 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:16.245492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" event={"ID":"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469","Type":"ContainerDied","Data":"6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866"}
Sep 16 01:05:16.245929 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:16.245887    2581 scope.go:117] "RemoveContainer" containerID="6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866"
Sep 16 01:05:16.247062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:16.247035    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:05:16.247134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:16.247121    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:05:16.247173 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:16.247154    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:05:17.253500 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:17.253436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" event={"ID":"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469","Type":"ContainerStarted","Data":"a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9"}
Sep 16 01:05:17.440199 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:17.440164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:17.671749 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:17.671697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:05:17.673063 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:17.673023    2581 patch_prober.go:28] interesting pod/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:05:17.673297 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:17.673265    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 16 01:05:18.250436 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:18.250401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:18.666414 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:18.666369    2581 patch_prober.go:28] interesting pod/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:05:18.667522 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:18.667062    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 16 01:05:18.972448 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:18.972352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:19.628526 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:19.628472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:19.666428 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:19.666382    2581 patch_prober.go:28] interesting pod/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:05:19.666966 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:19.666449    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 16 01:05:20.311552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:20.311513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:20.666744 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:20.666178    2581 patch_prober.go:28] interesting pod/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:05:20.666744 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:20.666264    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 16 01:05:21.667019 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:21.666979    2581 patch_prober.go:28] interesting pod/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:05:21.667654 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:21.667621    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 16 01:05:22.097269 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:22.097229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:05:22.670924 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:22.670885    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:05:22.672009 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:22.671990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:05:23.223731 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:23.223702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:24.012828 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:24.012745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:24.670049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:24.670016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:25.517164 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:25.517127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:25.768547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:25.768423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:25.868985 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:25.868952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:26.312708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:26.312672    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:27.081580 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:27.081542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:27.820701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:27.820654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:28.525324 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:28.525290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:29.269677 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:29.269646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:30.877747 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:30.877717    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:32.994148 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:32.994113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:33.195941 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.195910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:33.624425 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.624388    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"]
Sep 16 01:05:33.624712 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.624682    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss" podUID="4be65ea6-31eb-4102-a87b-380ec97fc374" containerName="maas-api" containerID="cri-o://f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4" gracePeriod=30
Sep 16 01:05:33.675166 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.675137    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:33.824183 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.824134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:33.875379 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.875312    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:05:33.899389 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.899334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:33.963815 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.963778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:33.970885 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.970854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s82w4\" (UniqueName: \"kubernetes.io/projected/4be65ea6-31eb-4102-a87b-380ec97fc374-kube-api-access-s82w4\") pod \"4be65ea6-31eb-4102-a87b-380ec97fc374\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") "
Sep 16 01:05:33.971033 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.970894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls\") pod \"4be65ea6-31eb-4102-a87b-380ec97fc374\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") "
Sep 16 01:05:33.971033 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.970986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-metrics-certs\") pod \"4be65ea6-31eb-4102-a87b-380ec97fc374\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") "
Sep 16 01:05:33.971142 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.971036    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be65ea6-31eb-4102-a87b-380ec97fc374-openshift-service-ca\") pod \"4be65ea6-31eb-4102-a87b-380ec97fc374\" (UID: \"4be65ea6-31eb-4102-a87b-380ec97fc374\") "
Sep 16 01:05:33.971372 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.971341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be65ea6-31eb-4102-a87b-380ec97fc374-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4be65ea6-31eb-4102-a87b-380ec97fc374" (UID: "4be65ea6-31eb-4102-a87b-380ec97fc374"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:33.971507 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.971414    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be65ea6-31eb-4102-a87b-380ec97fc374-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:33.973401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.973370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4be65ea6-31eb-4102-a87b-380ec97fc374" (UID: "4be65ea6-31eb-4102-a87b-380ec97fc374"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:33.973401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.973386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be65ea6-31eb-4102-a87b-380ec97fc374-kube-api-access-s82w4" (OuterVolumeSpecName: "kube-api-access-s82w4") pod "4be65ea6-31eb-4102-a87b-380ec97fc374" (UID: "4be65ea6-31eb-4102-a87b-380ec97fc374"). InnerVolumeSpecName "kube-api-access-s82w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:33.973604 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:33.973398    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4be65ea6-31eb-4102-a87b-380ec97fc374" (UID: "4be65ea6-31eb-4102-a87b-380ec97fc374"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:34.028290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.028243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:34.072652 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.072617    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82w4\" (UniqueName: \"kubernetes.io/projected/4be65ea6-31eb-4102-a87b-380ec97fc374-kube-api-access-s82w4\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:34.072652 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.072649    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:34.072652 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.072658    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4be65ea6-31eb-4102-a87b-380ec97fc374-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:34.102669 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.102627    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"]
Sep 16 01:05:34.102950 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.102916    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy" containerID="cri-o://a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9" gracePeriod=30
Sep 16 01:05:34.125633 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.125535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:34.366330 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.366291    2581 generic.go:356] "Generic (PLEG): container finished" podID="4be65ea6-31eb-4102-a87b-380ec97fc374" containerID="f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4" exitCode=0
Sep 16 01:05:34.366525 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.366342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss" event={"ID":"4be65ea6-31eb-4102-a87b-380ec97fc374","Type":"ContainerDied","Data":"f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4"}
Sep 16 01:05:34.366525 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.366403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss" event={"ID":"4be65ea6-31eb-4102-a87b-380ec97fc374","Type":"ContainerDied","Data":"323b6505834f6dfb063839462ed4d1352f31ccd8edcd0c6239cc1b2e980119ad"}
Sep 16 01:05:34.366525 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.366344    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"
Sep 16 01:05:34.366525 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.366423    2581 scope.go:117] "RemoveContainer" containerID="f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4"
Sep 16 01:05:34.376455 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.376438    2581 scope.go:117] "RemoveContainer" containerID="f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4"
Sep 16 01:05:34.376748 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:05:34.376729    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4\": container with ID starting with f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4 not found: ID does not exist" containerID="f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4"
Sep 16 01:05:34.376801 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.376754    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4"} err="failed to get container status \"f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4\": rpc error: code = NotFound desc = could not find container \"f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4\": container with ID starting with f2ca054656782fe2f35f2c9e3f8d2fa8dae1fd659db270f1d94080d3bad487a4 not found: ID does not exist"
Sep 16 01:05:34.386149 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.386125    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"]
Sep 16 01:05:34.389762 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.389735    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f097ba39-676bd554f9-dw8ss"]
Sep 16 01:05:34.515042 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.515008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:34.638271 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.638167    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:34.733803 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.733771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:34.885663 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:34.885630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:35.230828 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.230793    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"]
Sep 16 01:05:35.231323 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.231304    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6d59bf-2cd7-49e6-816e-9ff9e30c985a" containerName="main"
Sep 16 01:05:35.231406 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.231325    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6d59bf-2cd7-49e6-816e-9ff9e30c985a" containerName="main"
Sep 16 01:05:35.231406 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.231341    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be65ea6-31eb-4102-a87b-380ec97fc374" containerName="maas-api"
Sep 16 01:05:35.231406 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.231348    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be65ea6-31eb-4102-a87b-380ec97fc374" containerName="maas-api"
Sep 16 01:05:35.231579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.231428    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6d59bf-2cd7-49e6-816e-9ff9e30c985a" containerName="main"
Sep 16 01:05:35.231579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.231440    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be65ea6-31eb-4102-a87b-380ec97fc374" containerName="maas-api"
Sep 16 01:05:35.235986 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.235953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.238276 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.238254    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a58c3d9e-gw-openshift-default-dockercfg-ljvmg\""
Sep 16 01:05:35.246822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.246796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"]
Sep 16 01:05:35.264602 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.264567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:35.283493 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-socket\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283693 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-podinfo\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-token\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283786 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-certs\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283786 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-envoy\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283786 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2843ba3a-35c4-485f-b793-d4f22e1025ad-istiod-ca-cert\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283907 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-data\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283907 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8ls\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-kube-api-access-4z8ls\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.283907 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.283863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-credential-socket\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-data\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385741 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8ls\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-kube-api-access-4z8ls\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385741 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-credential-socket\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385741 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-socket\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-podinfo\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-token\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-certs\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-envoy\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.385918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2843ba3a-35c4-485f-b793-d4f22e1025ad-istiod-ca-cert\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.386234 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.385939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-data\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.386234 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.386131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-socket\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.386234 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.386147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-credential-socket\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.386435 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.386412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-certs\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.386785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.386765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2843ba3a-35c4-485f-b793-d4f22e1025ad-istiod-ca-cert\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.388246 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.388226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-envoy\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.388512 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.388492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-podinfo\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.397708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.397686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8ls\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-kube-api-access-4z8ls\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.400094 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.400065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-token\") pod \"e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") " pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.447135 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.447084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:35.548411 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.548375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:35.692285 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.692253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:35.827926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.827898    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"]
Sep 16 01:05:35.829840 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:05:35.829806    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2843ba3a_35c4_485f_b793_d4f22e1025ad.slice/crio-2277196b3daac2fff7d22bf9a3bde963400cf7ecc19f2d52bd7c012bb16b5665 WatchSource:0}: Error finding container 2277196b3daac2fff7d22bf9a3bde963400cf7ecc19f2d52bd7c012bb16b5665: Status 404 returned error can't find the container with id 2277196b3daac2fff7d22bf9a3bde963400cf7ecc19f2d52bd7c012bb16b5665
Sep 16 01:05:35.832374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.832341    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:05:35.832466 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.832409    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:05:35.832466 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.832444    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:05:35.942773 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:35.942743    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be65ea6-31eb-4102-a87b-380ec97fc374" path="/var/lib/kubelet/pods/4be65ea6-31eb-4102-a87b-380ec97fc374/volumes"
Sep 16 01:05:36.010126 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.010089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:36.079727 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.079617    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:36.173552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.173511    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:36.252742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.252631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:36.314759 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.314720    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:36.383121 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.383025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" event={"ID":"2843ba3a-35c4-485f-b793-d4f22e1025ad","Type":"ContainerStarted","Data":"a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447"}
Sep 16 01:05:36.383121 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.383085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" event={"ID":"2843ba3a-35c4-485f-b793-d4f22e1025ad","Type":"ContainerStarted","Data":"2277196b3daac2fff7d22bf9a3bde963400cf7ecc19f2d52bd7c012bb16b5665"}
Sep 16 01:05:36.405432 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.405368    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" podStartSLOduration=1.405346292 podStartE2EDuration="1.405346292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:05:36.403258102 +0000 UTC m=+1354.946395088" watchObservedRunningTime="2026-09-16 01:05:36.405346292 +0000 UTC m=+1354.948483268"
Sep 16 01:05:36.549575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.549528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:36.554188 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.554162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:36.647046 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:36.646955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:37.319971 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:37.319937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:37.390853 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:37.390816    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:37.391687 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:37.391665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:05:37.673732 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:37.673651    2581 patch_prober.go:28] interesting pod/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 01:05:37.673732 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:37.673709    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 01:05:37.972915 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:37.972820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:38.087988 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:38.087952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:38.207884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:38.207846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:38.686072 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:38.686039    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:05:38.983505 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:38.983398    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"]
Sep 16 01:05:38.988466 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:38.988437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:38.990802 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:38.990773    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a58c3d9e-serving-cert\""
Sep 16 01:05:38.996309 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:38.996277    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"]
Sep 16 01:05:39.014110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.014052    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:39.114630 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.114598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:39.125977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.125939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.126136 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.125991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-metrics-certs\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.126136 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.126102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qsww\" (UniqueName: \"kubernetes.io/projected/85027f58-92e5-4ec7-97a2-04c9c7d21346-kube-api-access-8qsww\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.126220 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.126172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85027f58-92e5-4ec7-97a2-04c9c7d21346-openshift-service-ca\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.227630 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.227599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85027f58-92e5-4ec7-97a2-04c9c7d21346-openshift-service-ca\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.227760 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.227716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.227760 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.227746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-metrics-certs\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.227878 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.227786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qsww\" (UniqueName: \"kubernetes.io/projected/85027f58-92e5-4ec7-97a2-04c9c7d21346-kube-api-access-8qsww\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.227936 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:05:39.227880    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-serving-cert: secret "maas-api-e2e-migrate-a58c3d9e-serving-cert" not found
Sep 16 01:05:39.227990 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:05:39.227959    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls podName:85027f58-92e5-4ec7-97a2-04c9c7d21346 nodeName:}" failed. No retries permitted until 2026-09-16 01:05:39.727937351 +0000 UTC m=+1358.271074312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls") pod "maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh" (UID: "85027f58-92e5-4ec7-97a2-04c9c7d21346") : secret "maas-api-e2e-migrate-a58c3d9e-serving-cert" not found
Sep 16 01:05:39.228434 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.228409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85027f58-92e5-4ec7-97a2-04c9c7d21346-openshift-service-ca\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.230130 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.230109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-metrics-certs\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.236803 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.236731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qsww\" (UniqueName: \"kubernetes.io/projected/85027f58-92e5-4ec7-97a2-04c9c7d21346-kube-api-access-8qsww\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.380636 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.380607    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:05:39.405682 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.405644    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerID="a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9" exitCode=0
Sep 16 01:05:39.405871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.405708    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"
Sep 16 01:05:39.405871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.405720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" event={"ID":"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469","Type":"ContainerDied","Data":"a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9"}
Sep 16 01:05:39.405871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.405779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p" event={"ID":"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469","Type":"ContainerDied","Data":"3e868b2ffff528c158c3c94ee5e97081e82317b33af4332b0480f14fec0467b9"}
Sep 16 01:05:39.405871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.405803    2581 scope.go:117] "RemoveContainer" containerID="a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9"
Sep 16 01:05:39.415869 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.415843    2581 scope.go:117] "RemoveContainer" containerID="6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866"
Sep 16 01:05:39.427264 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.427243    2581 scope.go:117] "RemoveContainer" containerID="a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9"
Sep 16 01:05:39.427624 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:05:39.427597    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9\": container with ID starting with a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9 not found: ID does not exist" containerID="a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9"
Sep 16 01:05:39.427723 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.427633    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9"} err="failed to get container status \"a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9\": rpc error: code = NotFound desc = could not find container \"a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9\": container with ID starting with a38b718f1466282cdd5bbd095a92469ae3756afcb3b3fc6f2f4613aefed1b6a9 not found: ID does not exist"
Sep 16 01:05:39.427723 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.427652    2581 scope.go:117] "RemoveContainer" containerID="6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866"
Sep 16 01:05:39.427953 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:05:39.427936    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866\": container with ID starting with 6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866 not found: ID does not exist" containerID="6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866"
Sep 16 01:05:39.427996 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.427957    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866"} err="failed to get container status \"6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866\": rpc error: code = NotFound desc = could not find container \"6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866\": container with ID starting with 6c5cb726090f572ca8a910c55b4f981ad8a798b69d037a919df35d5d2bd1a866 not found: ID does not exist"
Sep 16 01:05:39.430313 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-token\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430391 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-envoy\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430431 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-certs\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430431 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf57m\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-kube-api-access-xf57m\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430546 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-podinfo\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430546 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-data\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430650 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istiod-ca-cert\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430650 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-socket\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430752 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-credential-socket\") pod \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\" (UID: \"e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469\") "
Sep 16 01:05:39.430805 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430746    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:39.431009 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.430949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:39.431093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.431027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:39.431093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.431083    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.431205 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.431102    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.431261 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.431198    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:39.431261 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.431226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:39.433110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.433074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-kube-api-access-xf57m" (OuterVolumeSpecName: "kube-api-access-xf57m") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "kube-api-access-xf57m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:39.433214 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.433192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:39.433268 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.433232    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:39.433327 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.433301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" (UID: "e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:05:39.485961 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.485924    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:39.532678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.532646    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.532678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.532675    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.532678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.532685    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.532926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.532693    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf57m\" (UniqueName: \"kubernetes.io/projected/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-kube-api-access-xf57m\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.532926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.532703    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.532926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.532711    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.532926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.532718    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:39.585592 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.585558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:39.683656 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.683612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:39.730496 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.730449    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"]
Sep 16 01:05:39.734694 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.734666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.736691 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.736672    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f097ba39-openshift-default-7dbd7dc76b-lnv5p"]
Sep 16 01:05:39.737150 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.737132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls\") pod \"maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.905263 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.905166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:39.941862 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:39.941830    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" path="/var/lib/kubelet/pods/e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469/volumes"
Sep 16 01:05:40.019335 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:40.019306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:40.178110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:40.178078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"]
Sep 16 01:05:40.179761 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:05:40.179735    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85027f58_92e5_4ec7_97a2_04c9c7d21346.slice/crio-ed0508da2bd2af598fcb8bd03b94c5fcfb76a24785f73ca68c0b5398bbfba77b WatchSource:0}: Error finding container ed0508da2bd2af598fcb8bd03b94c5fcfb76a24785f73ca68c0b5398bbfba77b: Status 404 returned error can't find the container with id ed0508da2bd2af598fcb8bd03b94c5fcfb76a24785f73ca68c0b5398bbfba77b
Sep 16 01:05:40.413365 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:40.413329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh" event={"ID":"85027f58-92e5-4ec7-97a2-04c9c7d21346","Type":"ContainerStarted","Data":"ed0508da2bd2af598fcb8bd03b94c5fcfb76a24785f73ca68c0b5398bbfba77b"}
Sep 16 01:05:40.805645 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:40.805613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:41.420182 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:41.420143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh" event={"ID":"85027f58-92e5-4ec7-97a2-04c9c7d21346","Type":"ContainerStarted","Data":"7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150"}
Sep 16 01:05:41.420356 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:41.420322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:41.436450 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:41.436397    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh" podStartSLOduration=2.90946128 podStartE2EDuration="3.436353791s" totalImagesPullingTime="526.892511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:05:41.435148178 +0000 UTC m=+1359.978285148" watchObservedRunningTime="2026-09-16 01:05:41.436353791 +0000 UTC m=+1359.979490760"
Sep 16 01:05:42.587899 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:42.587865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:42.689365 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:42.689318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:42.851159 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:42.851073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:43.533328 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:43.533288    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:44.422419 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:44.422382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:45.126849 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:45.126814    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:45.801791 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:45.801757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:46.763915 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:46.763883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:47.431134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:47.431104    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:05:53.737714 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:53.737679    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"]
Sep 16 01:05:53.738142 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:53.737899    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp" podUID="ddc43567-c96f-4200-870a-007ab24d462e" containerName="maas-api" containerID="cri-o://d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab" gracePeriod=30
Sep 16 01:05:53.769773 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:53.769738    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:53.842965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:53.842934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:53.904244 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:53.904207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:53.969972 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:53.969934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:53.999544 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:53.999455    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:05:54.027636 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.027595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:54.184745 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.184709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls\") pod \"ddc43567-c96f-4200-870a-007ab24d462e\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") "
Sep 16 01:05:54.184918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.184776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-metrics-certs\") pod \"ddc43567-c96f-4200-870a-007ab24d462e\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") "
Sep 16 01:05:54.184918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.184816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p8qc\" (UniqueName: \"kubernetes.io/projected/ddc43567-c96f-4200-870a-007ab24d462e-kube-api-access-5p8qc\") pod \"ddc43567-c96f-4200-870a-007ab24d462e\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") "
Sep 16 01:05:54.184918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.184863    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc43567-c96f-4200-870a-007ab24d462e-openshift-service-ca\") pod \"ddc43567-c96f-4200-870a-007ab24d462e\" (UID: \"ddc43567-c96f-4200-870a-007ab24d462e\") "
Sep 16 01:05:54.185293 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.185263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc43567-c96f-4200-870a-007ab24d462e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ddc43567-c96f-4200-870a-007ab24d462e" (UID: "ddc43567-c96f-4200-870a-007ab24d462e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:54.186974 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.186937    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ddc43567-c96f-4200-870a-007ab24d462e" (UID: "ddc43567-c96f-4200-870a-007ab24d462e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:54.187095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.186993    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc43567-c96f-4200-870a-007ab24d462e-kube-api-access-5p8qc" (OuterVolumeSpecName: "kube-api-access-5p8qc") pod "ddc43567-c96f-4200-870a-007ab24d462e" (UID: "ddc43567-c96f-4200-870a-007ab24d462e"). InnerVolumeSpecName "kube-api-access-5p8qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:54.187095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.186989    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ddc43567-c96f-4200-870a-007ab24d462e" (UID: "ddc43567-c96f-4200-870a-007ab24d462e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:54.286032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.285991    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:54.286032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.286027    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc43567-c96f-4200-870a-007ab24d462e-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:54.286032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.286037    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8qc\" (UniqueName: \"kubernetes.io/projected/ddc43567-c96f-4200-870a-007ab24d462e-kube-api-access-5p8qc\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:54.286290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.286048    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc43567-c96f-4200-870a-007ab24d462e-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:05:54.336602 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.336560    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:54.436769 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.436735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:54.490326 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.490295    2581 generic.go:356] "Generic (PLEG): container finished" podID="ddc43567-c96f-4200-870a-007ab24d462e" containerID="d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab" exitCode=0
Sep 16 01:05:54.490536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.490359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp" event={"ID":"ddc43567-c96f-4200-870a-007ab24d462e","Type":"ContainerDied","Data":"d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab"}
Sep 16 01:05:54.490536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.490385    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"
Sep 16 01:05:54.490536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.490435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp" event={"ID":"ddc43567-c96f-4200-870a-007ab24d462e","Type":"ContainerDied","Data":"0869739d350a2db974e19ae88e9eb1f82ee7381e469e88c29daf7f9e63ccb34e"}
Sep 16 01:05:54.490536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.490465    2581 scope.go:117] "RemoveContainer" containerID="d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab"
Sep 16 01:05:54.500361 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.500340    2581 scope.go:117] "RemoveContainer" containerID="d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab"
Sep 16 01:05:54.500661 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:05:54.500637    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab\": container with ID starting with d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab not found: ID does not exist" containerID="d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab"
Sep 16 01:05:54.500733 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.500669    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab"} err="failed to get container status \"d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab\": rpc error: code = NotFound desc = could not find container \"d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab\": container with ID starting with d40f8aab5ff0b151638d8537c9d90cc38f832bd39465298cf18dfa438ac142ab not found: ID does not exist"
Sep 16 01:05:54.511252 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.511221    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"]
Sep 16 01:05:54.514924 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:54.514897    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d0d72-5b6b4cc4fc-t4tfp"]
Sep 16 01:05:55.134969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:55.134931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:55.941133 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:55.941088    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc43567-c96f-4200-870a-007ab24d462e" path="/var/lib/kubelet/pods/ddc43567-c96f-4200-870a-007ab24d462e/volumes"
Sep 16 01:05:56.198283 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:56.198188    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:56.860932 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:56.860890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:57.831258 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:57.831214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:58.711093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:58.711060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:05:59.413982 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:05:59.413946    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:00.283976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:00.283937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:00.385147 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:00.385101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:00.587548 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:00.587428    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"]
Sep 16 01:06:00.587936 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:00.587758    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" containerID="cri-o://c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc" gracePeriod=30
Sep 16 01:06:00.833742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:00.833708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:01.124892 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:01.124861    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:01.227198 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:01.227163    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:01.401193 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:01.401107    2581 patch_prober.go:28] interesting pod/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 01:06:01.401350 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:01.401219    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 01:06:01.465731 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:01.465684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:01.772110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:01.772072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:03.714404 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:03.714372    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:04.183452 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:04.183411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:04.268955 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:04.268919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:04.604303 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:04.604271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:05.722754 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.722714    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:05.863836 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.863762    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:06:05.902505 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.902440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-certs\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.902705 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.902551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-data\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.902705 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.902587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-podinfo\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.902705 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.902627    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f88175-8d05-4b62-9a99-56116a0549b3-istiod-ca-cert\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.902705 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.902691    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-credential-socket\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.902926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.902751    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-envoy\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.902926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.902766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:05.903034 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:05.903034 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903024    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:05.903152 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903141    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f88175-8d05-4b62-9a99-56116a0549b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:06:05.903247 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-socket\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.903307 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-kube-api-access-8w8tx\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.903365 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903308    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-token\") pod \"f8f88175-8d05-4b62-9a99-56116a0549b3\" (UID: \"f8f88175-8d05-4b62-9a99-56116a0549b3\") "
Sep 16 01:06:05.903507 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:05.903785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903759    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:05.903991 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903972    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:05.904098 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.903994    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f88175-8d05-4b62-9a99-56116a0549b3-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:05.904098 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.904008    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:05.904098 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.904023    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:05.905731 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.905701    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:06:05.905930 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.905904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:05.906143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.906118    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-kube-api-access-8w8tx" (OuterVolumeSpecName: "kube-api-access-8w8tx") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "kube-api-access-8w8tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:05.906345 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:05.906328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8f88175-8d05-4b62-9a99-56116a0549b3" (UID: "f8f88175-8d05-4b62-9a99-56116a0549b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:06.005063 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.005024    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-kube-api-access-8w8tx\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:06.005063 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.005057    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:06.005063 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.005068    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:06.005359 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.005078    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f88175-8d05-4b62-9a99-56116a0549b3-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:06.557967 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.557925    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerID="c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc" exitCode=0
Sep 16 01:06:06.558159 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.557981    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"
Sep 16 01:06:06.558159 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.557989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" event={"ID":"f8f88175-8d05-4b62-9a99-56116a0549b3","Type":"ContainerDied","Data":"c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc"}
Sep 16 01:06:06.558159 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.558052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z" event={"ID":"f8f88175-8d05-4b62-9a99-56116a0549b3","Type":"ContainerDied","Data":"5e6707c377b0f485008a63f44fa6fbf4b7c1a29e93ec1fdff393ae145e7156ba"}
Sep 16 01:06:06.558159 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.558088    2581 scope.go:117] "RemoveContainer" containerID="c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc"
Sep 16 01:06:06.567466 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.567447    2581 scope.go:117] "RemoveContainer" containerID="17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8"
Sep 16 01:06:06.577832 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.577511    2581 scope.go:117] "RemoveContainer" containerID="c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc"
Sep 16 01:06:06.577832 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.577606    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"]
Sep 16 01:06:06.580261 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:06.580191    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc\": container with ID starting with c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc not found: ID does not exist" containerID="c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc"
Sep 16 01:06:06.580261 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.580233    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc"} err="failed to get container status \"c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc\": rpc error: code = NotFound desc = could not find container \"c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc\": container with ID starting with c77e446107fbf80ab020dddef19bedeeef2834ec6e1dfa3f7c1f824f1ad7dbdc not found: ID does not exist"
Sep 16 01:06:06.580261 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.580258    2581 scope.go:117] "RemoveContainer" containerID="17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8"
Sep 16 01:06:06.580708 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:06.580678    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8\": container with ID starting with 17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8 not found: ID does not exist" containerID="17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8"
Sep 16 01:06:06.580827 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.580716    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8"} err="failed to get container status \"17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8\": rpc error: code = NotFound desc = could not find container \"17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8\": container with ID starting with 17c142e6d0b4485a06dd4d11b6a16e86e9af9d85287e2cbf80c204a732f627a8 not found: ID does not exist"
Sep 16 01:06:06.581629 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.581610    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6d0d72-openshift-default-65f77f5774-2rm5z"]
Sep 16 01:06:06.675823 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.675789    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"]
Sep 16 01:06:06.676386 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676369    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy"
Sep 16 01:06:06.676502 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy"
Sep 16 01:06:06.676502 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676402    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy"
Sep 16 01:06:06.676502 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676411    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy"
Sep 16 01:06:06.676502 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676426    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc43567-c96f-4200-870a-007ab24d462e" containerName="maas-api"
Sep 16 01:06:06.676502 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc43567-c96f-4200-870a-007ab24d462e" containerName="maas-api"
Sep 16 01:06:06.676502 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676464    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy"
Sep 16 01:06:06.676502 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676474    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy"
Sep 16 01:06:06.676888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676506    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy"
Sep 16 01:06:06.676888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676515    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy"
Sep 16 01:06:06.676888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676606    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy"
Sep 16 01:06:06.676888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676617    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" containerName="istio-proxy"
Sep 16 01:06:06.676888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676631    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy"
Sep 16 01:06:06.676888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676640    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c2140a-8a33-4bf7-96b7-fe8f4b9dc469" containerName="istio-proxy"
Sep 16 01:06:06.676888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.676652    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc43567-c96f-4200-870a-007ab24d462e" containerName="maas-api"
Sep 16 01:06:06.681658 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.681634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.685015 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.684799    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f831c5-serving-cert\""
Sep 16 01:06:06.687769 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.687700    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"]
Sep 16 01:06:06.706609 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.706579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:06.767191 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.767142    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:06.813972 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.813874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.813972 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.813926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrjf\" (UniqueName: \"kubernetes.io/projected/4a0f295f-fbea-4d21-994b-746140b11108-kube-api-access-snrjf\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.814189 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.814028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f295f-fbea-4d21-994b-746140b11108-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.814189 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.814074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.915578 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.915530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.915578 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.915580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrjf\" (UniqueName: \"kubernetes.io/projected/4a0f295f-fbea-4d21-994b-746140b11108-kube-api-access-snrjf\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.915835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.915611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f295f-fbea-4d21-994b-746140b11108-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.915835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.915641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.915835 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:06.915682    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-serving-cert: secret "maas-api-e2e-shared-a-w2-f831c5-serving-cert" not found
Sep 16 01:06:06.915835 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:06.915762    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls podName:4a0f295f-fbea-4d21-994b-746140b11108 nodeName:}" failed. No retries permitted until 2026-09-16 01:06:07.415744136 +0000 UTC m=+1385.958881090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls") pod "maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" (UID: "4a0f295f-fbea-4d21-994b-746140b11108") : secret "maas-api-e2e-shared-a-w2-f831c5-serving-cert" not found
Sep 16 01:06:06.916505 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.916459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f295f-fbea-4d21-994b-746140b11108-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.918201 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.918170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:06.923702 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:06.923673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrjf\" (UniqueName: \"kubernetes.io/projected/4a0f295f-fbea-4d21-994b-746140b11108-kube-api-access-snrjf\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:07.005464 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.005352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:07.176408 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.176321    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:07.270967 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.270931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:07.420898 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.420857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:07.423374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.423351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:07.598064 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.598023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:07.625977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.625935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:07.874946 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.874917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"]
Sep 16 01:06:07.876826 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:06:07.876783    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0f295f_fbea_4d21_994b_746140b11108.slice/crio-75986e05978478f68cb2088854175cf9fdf8f95abc128b63225e747cdc67e390 WatchSource:0}: Error finding container 75986e05978478f68cb2088854175cf9fdf8f95abc128b63225e747cdc67e390: Status 404 returned error can't find the container with id 75986e05978478f68cb2088854175cf9fdf8f95abc128b63225e747cdc67e390
Sep 16 01:06:07.941835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:07.941799    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f88175-8d05-4b62-9a99-56116a0549b3" path="/var/lib/kubelet/pods/f8f88175-8d05-4b62-9a99-56116a0549b3/volumes"
Sep 16 01:06:08.012136 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.012101    2581 patch_prober.go:28] interesting pod/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:35904->10.132.0.51:15021: read: connection reset by peer" start-of-body=
Sep 16 01:06:08.012279 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.012167    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:35904->10.132.0.51:15021: read: connection reset by peer"
Sep 16 01:06:08.571936 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.571895    2581 generic.go:356] "Generic (PLEG): container finished" podID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerID="a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447" exitCode=0
Sep 16 01:06:08.572135 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.571978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" event={"ID":"2843ba3a-35c4-485f-b793-d4f22e1025ad","Type":"ContainerDied","Data":"a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447"}
Sep 16 01:06:08.572547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.572525    2581 scope.go:117] "RemoveContainer" containerID="a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447"
Sep 16 01:06:08.573877 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.573838    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:08.573990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.573930    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:08.573990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.573979    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:08.574198 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.574171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" event={"ID":"4a0f295f-fbea-4d21-994b-746140b11108","Type":"ContainerStarted","Data":"a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e"}
Sep 16 01:06:08.574320 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.574218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" event={"ID":"4a0f295f-fbea-4d21-994b-746140b11108","Type":"ContainerStarted","Data":"75986e05978478f68cb2088854175cf9fdf8f95abc128b63225e747cdc67e390"}
Sep 16 01:06:08.574388 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.574321    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:08.575260 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.575240    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:08.605723 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:08.605650    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" podStartSLOduration=2.250951156 podStartE2EDuration="2.605628873s" totalImagesPullingTime="354.677717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:08.605092104 +0000 UTC m=+1387.148229077" watchObservedRunningTime="2026-09-16 01:06:08.605628873 +0000 UTC m=+1387.148765848"
Sep 16 01:06:09.110693 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.110659    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"]
Sep 16 01:06:09.111223 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.110889    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh" podUID="85027f58-92e5-4ec7-97a2-04c9c7d21346" containerName="maas-api" containerID="cri-o://7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150" gracePeriod=30
Sep 16 01:06:09.139374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.139335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:09.239658 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.239629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:09.343012 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.342979    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:09.360073 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.360047    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:06:09.440906 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.440817    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qsww\" (UniqueName: \"kubernetes.io/projected/85027f58-92e5-4ec7-97a2-04c9c7d21346-kube-api-access-8qsww\") pod \"85027f58-92e5-4ec7-97a2-04c9c7d21346\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") "
Sep 16 01:06:09.440906 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.440861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85027f58-92e5-4ec7-97a2-04c9c7d21346-openshift-service-ca\") pod \"85027f58-92e5-4ec7-97a2-04c9c7d21346\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") "
Sep 16 01:06:09.441152 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.440927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls\") pod \"85027f58-92e5-4ec7-97a2-04c9c7d21346\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") "
Sep 16 01:06:09.441152 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.441035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-metrics-certs\") pod \"85027f58-92e5-4ec7-97a2-04c9c7d21346\" (UID: \"85027f58-92e5-4ec7-97a2-04c9c7d21346\") "
Sep 16 01:06:09.441584 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.441554    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85027f58-92e5-4ec7-97a2-04c9c7d21346-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85027f58-92e5-4ec7-97a2-04c9c7d21346" (UID: "85027f58-92e5-4ec7-97a2-04c9c7d21346"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:06:09.443345 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.443319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85027f58-92e5-4ec7-97a2-04c9c7d21346" (UID: "85027f58-92e5-4ec7-97a2-04c9c7d21346"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:06:09.443451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.443341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "85027f58-92e5-4ec7-97a2-04c9c7d21346" (UID: "85027f58-92e5-4ec7-97a2-04c9c7d21346"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:06:09.443646 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.443630    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85027f58-92e5-4ec7-97a2-04c9c7d21346-kube-api-access-8qsww" (OuterVolumeSpecName: "kube-api-access-8qsww") pod "85027f58-92e5-4ec7-97a2-04c9c7d21346" (UID: "85027f58-92e5-4ec7-97a2-04c9c7d21346"). InnerVolumeSpecName "kube-api-access-8qsww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:09.465572 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.465537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:09.509613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.509578    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"]
Sep 16 01:06:09.531540 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.531499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:09.542836 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.542806    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qsww\" (UniqueName: \"kubernetes.io/projected/85027f58-92e5-4ec7-97a2-04c9c7d21346-kube-api-access-8qsww\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:09.542836 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.542834    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85027f58-92e5-4ec7-97a2-04c9c7d21346-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:09.542997 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.542849    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:09.542997 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.542861    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85027f58-92e5-4ec7-97a2-04c9c7d21346-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:09.580695 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.580665    2581 generic.go:356] "Generic (PLEG): container finished" podID="85027f58-92e5-4ec7-97a2-04c9c7d21346" containerID="7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150" exitCode=0
Sep 16 01:06:09.580883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.580713    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"
Sep 16 01:06:09.580883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.580750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh" event={"ID":"85027f58-92e5-4ec7-97a2-04c9c7d21346","Type":"ContainerDied","Data":"7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150"}
Sep 16 01:06:09.580883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.580802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh" event={"ID":"85027f58-92e5-4ec7-97a2-04c9c7d21346","Type":"ContainerDied","Data":"ed0508da2bd2af598fcb8bd03b94c5fcfb76a24785f73ca68c0b5398bbfba77b"}
Sep 16 01:06:09.580883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.580826    2581 scope.go:117] "RemoveContainer" containerID="7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150"
Sep 16 01:06:09.583200 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.583173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" event={"ID":"2843ba3a-35c4-485f-b793-d4f22e1025ad","Type":"ContainerStarted","Data":"c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b"}
Sep 16 01:06:09.583344 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.583297    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy" containerID="cri-o://c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b" gracePeriod=30
Sep 16 01:06:09.592657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.592626    2581 scope.go:117] "RemoveContainer" containerID="7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150"
Sep 16 01:06:09.593155 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:09.593127    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150\": container with ID starting with 7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150 not found: ID does not exist" containerID="7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150"
Sep 16 01:06:09.593254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.593168    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150"} err="failed to get container status \"7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150\": rpc error: code = NotFound desc = could not find container \"7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150\": container with ID starting with 7c74cf04f94e59f6a919c065792b87a18039db7d1fb477fa1c41ce0926201150 not found: ID does not exist"
Sep 16 01:06:09.615436 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.615389    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"]
Sep 16 01:06:09.618383 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.618348    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a58c3d9e-5856bc8f4f-j6bhh"]
Sep 16 01:06:09.942006 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:09.941963    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85027f58-92e5-4ec7-97a2-04c9c7d21346" path="/var/lib/kubelet/pods/85027f58-92e5-4ec7-97a2-04c9c7d21346/volumes"
Sep 16 01:06:10.014427 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.014391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:10.318000 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.317964    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"]
Sep 16 01:06:10.318604 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.318573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85027f58-92e5-4ec7-97a2-04c9c7d21346" containerName="maas-api"
Sep 16 01:06:10.318604 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.318596    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85027f58-92e5-4ec7-97a2-04c9c7d21346" containerName="maas-api"
Sep 16 01:06:10.318781 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.318726    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85027f58-92e5-4ec7-97a2-04c9c7d21346" containerName="maas-api"
Sep 16 01:06:10.322322 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.322293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.325195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.325172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-30fb702c-openshift-default-dockercfg-njxq5\""
Sep 16 01:06:10.346844 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.346815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"]
Sep 16 01:06:10.372508 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.372444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:10.451708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-data\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.451903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-socket\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.451903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istiod-ca-cert\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.451903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-certs\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.451903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-token\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.452041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-podinfo\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.452041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-envoy\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.452041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.451992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-credential-socket\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.452041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.452007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zsf\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-kube-api-access-l4zsf\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.470451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.470413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:10.553461 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-data\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.553700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-socket\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.553700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istiod-ca-cert\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.553700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-certs\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.553700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-token\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.553700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-podinfo\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.553700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-envoy\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.553700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-credential-socket\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.554078 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zsf\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-kube-api-access-l4zsf\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.554078 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.553996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-certs\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.554189 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.554170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-socket\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.554253 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.554192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-credential-socket\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.554329 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.554303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istiod-ca-cert\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.554384 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.554364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-data\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.556265 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.556245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-envoy\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.556467 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.556448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-podinfo\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.561284 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.561262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-token\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.561422 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.561403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zsf\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-kube-api-access-l4zsf\") pod \"e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") " pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.634709 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.634605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:10.818619 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.818580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:10.927769 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.927745    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"]
Sep 16 01:06:10.929938 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:06:10.929909    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c235c2_95e0_4afa_8db3_051f2e1ebf8b.slice/crio-55a210f4a54eb4e848f37e1ee5b90a279fdabc3057b4734fb41651f7aca3327e WatchSource:0}: Error finding container 55a210f4a54eb4e848f37e1ee5b90a279fdabc3057b4734fb41651f7aca3327e: Status 404 returned error can't find the container with id 55a210f4a54eb4e848f37e1ee5b90a279fdabc3057b4734fb41651f7aca3327e
Sep 16 01:06:10.932259 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.932227    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:10.932335 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.932291    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:10.932335 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:10.932324    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:11.235657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:11.235569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:11.387168 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:11.387130    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:06:11.597344 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:11.597300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" event={"ID":"65c235c2-95e0-4afa-8db3-051f2e1ebf8b","Type":"ContainerStarted","Data":"f68f6bb4798a91a147988a8cde482c9a4189b80706d0edbd4fb10f0f4ace233a"}
Sep 16 01:06:11.597537 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:11.597365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" event={"ID":"65c235c2-95e0-4afa-8db3-051f2e1ebf8b","Type":"ContainerStarted","Data":"55a210f4a54eb4e848f37e1ee5b90a279fdabc3057b4734fb41651f7aca3327e"}
Sep 16 01:06:11.617811 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:11.617758    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" podStartSLOduration=1.617740379 podStartE2EDuration="1.617740379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:11.614886114 +0000 UTC m=+1390.158023102" watchObservedRunningTime="2026-09-16 01:06:11.617740379 +0000 UTC m=+1390.160877348"
Sep 16 01:06:11.635451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:11.635414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:11.640846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:11.640813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:12.419061 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:12.419018    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:12.604152 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:12.604122    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:12.605092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:12.605070    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:12.987788 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:12.987758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:13.088567 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:13.088531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:13.398011 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:13.397969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:14.284646 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.284615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:14.588871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.588792    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:06:14.857431 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.857405    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:06:14.995531 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995463    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z8ls\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-kube-api-access-4z8ls\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.995531 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-envoy\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.995799 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-data\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.995799 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-socket\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.995799 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-podinfo\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.995799 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995684    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2843ba3a-35c4-485f-b793-d4f22e1025ad-istiod-ca-cert\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.995799 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-certs\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.996055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:14.996055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:14.996055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:14.996055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.995757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-token\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.996258 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2843ba3a-35c4-485f-b793-d4f22e1025ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:06:14.996258 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-credential-socket\") pod \"2843ba3a-35c4-485f-b793-d4f22e1025ad\" (UID: \"2843ba3a-35c4-485f-b793-d4f22e1025ad\") "
Sep 16 01:06:14.996400 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:14.996746 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996722    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:14.996847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996750    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:14.996847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996765    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2843ba3a-35c4-485f-b793-d4f22e1025ad-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:14.996847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996780    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:14.996847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.996795    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:14.997972 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.997945    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-kube-api-access-4z8ls" (OuterVolumeSpecName: "kube-api-access-4z8ls") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "kube-api-access-4z8ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:14.998088 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.997948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:06:14.998088 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.998036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:06:14.998289 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:14.998270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "2843ba3a-35c4-485f-b793-d4f22e1025ad" (UID: "2843ba3a-35c4-485f-b793-d4f22e1025ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:15.097917 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.097794    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:15.097917 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.097828    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:15.097917 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.097838    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8ls\" (UniqueName: \"kubernetes.io/projected/2843ba3a-35c4-485f-b793-d4f22e1025ad-kube-api-access-4z8ls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:15.097917 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.097848    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2843ba3a-35c4-485f-b793-d4f22e1025ad-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:15.404375 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.404260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:15.623588 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.623552    2581 generic.go:356] "Generic (PLEG): container finished" podID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerID="c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b" exitCode=0
Sep 16 01:06:15.623766 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.623610    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"
Sep 16 01:06:15.623766 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.623632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" event={"ID":"2843ba3a-35c4-485f-b793-d4f22e1025ad","Type":"ContainerDied","Data":"c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b"}
Sep 16 01:06:15.623766 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.623687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2" event={"ID":"2843ba3a-35c4-485f-b793-d4f22e1025ad","Type":"ContainerDied","Data":"2277196b3daac2fff7d22bf9a3bde963400cf7ecc19f2d52bd7c012bb16b5665"}
Sep 16 01:06:15.623766 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.623717    2581 scope.go:117] "RemoveContainer" containerID="c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b"
Sep 16 01:06:15.633698 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.633675    2581 scope.go:117] "RemoveContainer" containerID="a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447"
Sep 16 01:06:15.642383 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.642360    2581 scope.go:117] "RemoveContainer" containerID="c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b"
Sep 16 01:06:15.642707 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:15.642686    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b\": container with ID starting with c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b not found: ID does not exist" containerID="c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b"
Sep 16 01:06:15.642771 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.642718    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b"} err="failed to get container status \"c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b\": rpc error: code = NotFound desc = could not find container \"c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b\": container with ID starting with c0a5787e31d4fc1877ca090a4e5147029ab282d46143c2b85604f5fbc6b71c8b not found: ID does not exist"
Sep 16 01:06:15.642771 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.642738    2581 scope.go:117] "RemoveContainer" containerID="a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447"
Sep 16 01:06:15.643036 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:15.643007    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447\": container with ID starting with a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447 not found: ID does not exist" containerID="a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447"
Sep 16 01:06:15.643120 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.643042    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447"} err="failed to get container status \"a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447\": rpc error: code = NotFound desc = could not find container \"a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447\": container with ID starting with a02994592eb53cadd43a650d844e07ff039d44a0888644b38864930a2c870447 not found: ID does not exist"
Sep 16 01:06:15.645917 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.645894    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"]
Sep 16 01:06:15.649955 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.649933    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a58c3d9e-gw-openshift-default-7995cf64-c4km2"]
Sep 16 01:06:15.942344 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:15.942294    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" path="/var/lib/kubelet/pods/2843ba3a-35c4-485f-b793-d4f22e1025ad/volumes"
Sep 16 01:06:16.081746 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.081712    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"]
Sep 16 01:06:16.082614 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.082585    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy"
Sep 16 01:06:16.082614 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.082614    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy"
Sep 16 01:06:16.082761 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.082643    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy"
Sep 16 01:06:16.082761 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.082653    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy"
Sep 16 01:06:16.082761 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.082757    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy"
Sep 16 01:06:16.082866 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.082770    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2843ba3a-35c4-485f-b793-d4f22e1025ad" containerName="istio-proxy"
Sep 16 01:06:16.087884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.087863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.090544 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.090302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-30fb702c-serving-cert\""
Sep 16 01:06:16.093841 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.093815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"]
Sep 16 01:06:16.113756 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.113722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:16.167525 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.167491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:16.210844 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.210739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhngw\" (UniqueName: \"kubernetes.io/projected/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-kube-api-access-jhngw\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.210844 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.210799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-openshift-service-ca\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.211083 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.210892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-metrics-certs\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.211083 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.210936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-maas-api-tls\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.309231 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.309195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:16.312318 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.312295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhngw\" (UniqueName: \"kubernetes.io/projected/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-kube-api-access-jhngw\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.312390 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.312344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-openshift-service-ca\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.312390 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.312376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-metrics-certs\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.312503 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.312400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-maas-api-tls\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.312986 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.312964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-openshift-service-ca\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.314981 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.314957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-maas-api-tls\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.315068 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.314991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-metrics-certs\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.319215 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.319193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhngw\" (UniqueName: \"kubernetes.io/projected/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-kube-api-access-jhngw\") pod \"maas-api-e2e-ait-30fb702c-79f85656c-cp6dj\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.401629 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.401590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:16.676872 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.676844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"]
Sep 16 01:06:16.678600 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:06:16.678569    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f4380f_44f9_49ca_bafb_d035ab00a5d7.slice/crio-132cd006b137b854229ea01c0b08d1daefc2d794ffea88f12619c9d6912eb186 WatchSource:0}: Error finding container 132cd006b137b854229ea01c0b08d1daefc2d794ffea88f12619c9d6912eb186: Status 404 returned error can't find the container with id 132cd006b137b854229ea01c0b08d1daefc2d794ffea88f12619c9d6912eb186
Sep 16 01:06:16.752770 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.752732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:16.852994 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:16.852962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:17.314760 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:17.314728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:17.637131 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:17.637092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj" event={"ID":"f9f4380f-44f9-49ca-bafb-d035ab00a5d7","Type":"ContainerStarted","Data":"2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445"}
Sep 16 01:06:17.637322 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:17.637176    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj" event={"ID":"f9f4380f-44f9-49ca-bafb-d035ab00a5d7","Type":"ContainerStarted","Data":"132cd006b137b854229ea01c0b08d1daefc2d794ffea88f12619c9d6912eb186"}
Sep 16 01:06:17.637322 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:17.637214    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:17.652778 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:17.652727    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj" podStartSLOduration=1.062958389 podStartE2EDuration="1.652711335s" totalImagesPullingTime="589.752946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:17.651167888 +0000 UTC m=+1396.194304863" watchObservedRunningTime="2026-09-16 01:06:17.652711335 +0000 UTC m=+1396.195848309"
Sep 16 01:06:18.080435 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:18.080399    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:18.991942 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:18.991909    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:19.094686 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:19.094654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:19.340166 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:19.340132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:19.670807 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:19.670715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:19.796697 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:19.796646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:19.896735 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:19.896699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:21.693180 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:21.693138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:22.332273 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:22.332241    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:22.423704 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:22.423671    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:22.828036 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:22.828006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:23.631867 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:23.631831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:23.648753 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:23.648719    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:24.530603 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:24.530559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:25.179677 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.179637    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"]
Sep 16 01:06:25.183473 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.183451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.185702 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.185677    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-0ee1bf-serving-cert\""
Sep 16 01:06:25.195257 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.195230    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"]
Sep 16 01:06:25.215079 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.215045    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:25.301459 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.301418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.301459 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.301473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.301771 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.301607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8vl\" (UniqueName: \"kubernetes.io/projected/178f7896-2e17-4e15-a032-5577d31b98ea-kube-api-access-vd8vl\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.301771 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.301639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/178f7896-2e17-4e15-a032-5577d31b98ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.315692 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.315656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:25.402465 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.402430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8vl\" (UniqueName: \"kubernetes.io/projected/178f7896-2e17-4e15-a032-5577d31b98ea-kube-api-access-vd8vl\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.402465 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.402469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/178f7896-2e17-4e15-a032-5577d31b98ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.402711 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.402599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.402711 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.402624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.402800 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:25.402741    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-serving-cert: secret "maas-api-e2e-shared-b-w2-0ee1bf-serving-cert" not found
Sep 16 01:06:25.402842 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:25.402817    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls podName:178f7896-2e17-4e15-a032-5577d31b98ea nodeName:}" failed. No retries permitted until 2026-09-16 01:06:25.902795785 +0000 UTC m=+1404.445932751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls") pod "maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn" (UID: "178f7896-2e17-4e15-a032-5577d31b98ea") : secret "maas-api-e2e-shared-b-w2-0ee1bf-serving-cert" not found
Sep 16 01:06:25.403174 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.403144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/178f7896-2e17-4e15-a032-5577d31b98ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.405404 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.405379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.411724 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.411697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8vl\" (UniqueName: \"kubernetes.io/projected/178f7896-2e17-4e15-a032-5577d31b98ea-kube-api-access-vd8vl\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.447232 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.447146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:25.676731 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.676698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:25.776572 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.776527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:25.907785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.907734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:25.910184 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:25.910163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:26.097708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:26.097620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:26.229417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:26.229378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:26.369588 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:26.369556    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"]
Sep 16 01:06:26.370766 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:06:26.370741    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178f7896_2e17_4e15_a032_5577d31b98ea.slice/crio-a0fbde4d04c2d9d32b2aadcfe3ce44104c140144b809e5bd3232b389996571d4 WatchSource:0}: Error finding container a0fbde4d04c2d9d32b2aadcfe3ce44104c140144b809e5bd3232b389996571d4: Status 404 returned error can't find the container with id a0fbde4d04c2d9d32b2aadcfe3ce44104c140144b809e5bd3232b389996571d4
Sep 16 01:06:26.691131 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:26.691046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn" event={"ID":"178f7896-2e17-4e15-a032-5577d31b98ea","Type":"ContainerStarted","Data":"a0fbde4d04c2d9d32b2aadcfe3ce44104c140144b809e5bd3232b389996571d4"}
Sep 16 01:06:27.005965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:27.005875    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:27.699205 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:27.699172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn" event={"ID":"178f7896-2e17-4e15-a032-5577d31b98ea","Type":"ContainerStarted","Data":"1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189"}
Sep 16 01:06:27.699703 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:27.699266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:27.714883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:27.714830    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn" podStartSLOduration=2.316298197 podStartE2EDuration="2.714814354s" totalImagesPullingTime="398.516157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:27.713455309 +0000 UTC m=+1406.256592281" watchObservedRunningTime="2026-09-16 01:06:27.714814354 +0000 UTC m=+1406.257951329"
Sep 16 01:06:27.742665 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:27.742630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:33.710386 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:33.710352    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:06:37.313820 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:37.313779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:38.550338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:38.550301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:39.332559 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:39.332526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:40.001678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:40.001641    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:40.771847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:40.771815    2581 generic.go:356] "Generic (PLEG): container finished" podID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerID="f68f6bb4798a91a147988a8cde482c9a4189b80706d0edbd4fb10f0f4ace233a" exitCode=0
Sep 16 01:06:40.772060 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:40.771887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" event={"ID":"65c235c2-95e0-4afa-8db3-051f2e1ebf8b","Type":"ContainerDied","Data":"f68f6bb4798a91a147988a8cde482c9a4189b80706d0edbd4fb10f0f4ace233a"}
Sep 16 01:06:40.772336 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:40.772322    2581 scope.go:117] "RemoveContainer" containerID="f68f6bb4798a91a147988a8cde482c9a4189b80706d0edbd4fb10f0f4ace233a"
Sep 16 01:06:40.773577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:40.773552    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:40.773636 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:40.773615    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:40.773674 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:40.773647    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:41.101467 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:41.101378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:41.785857 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:41.785810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" event={"ID":"65c235c2-95e0-4afa-8db3-051f2e1ebf8b","Type":"ContainerStarted","Data":"505c24ad4bdd8bd01711fd3ebad29e64265fce2f7f08db6af79a8b6c4ce0a5c0"}
Sep 16 01:06:41.945849 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:41.945816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:42.272074 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:42.272033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:42.604842 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:42.604741    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:42.634984 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:42.634949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:42.636300 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:42.636274    2581 patch_prober.go:28] interesting pod/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 01:06:42.636400 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:42.636334    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 01:06:42.828690 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:42.828653    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:43.500444 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:43.500410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:43.635713 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:43.635678    2581 patch_prober.go:28] interesting pod/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 01:06:43.635889 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:43.635739    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 01:06:44.333179 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:44.333143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:44.635547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:44.635434    2581 patch_prober.go:28] interesting pod/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 01:06:44.635547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:44.635517    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 01:06:45.240107 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:45.240068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:45.635114 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:45.635078    2581 patch_prober.go:28] interesting pod/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 01:06:45.635303 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:45.635133    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 01:06:46.029854 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:46.029820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:46.635564 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:46.635527    2581 patch_prober.go:28] interesting pod/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 01:06:46.635732 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:46.635596    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 01:06:46.705328 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:46.705278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:47.406458 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:47.406422    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:47.639449 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:47.639414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:47.640370 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:47.640353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:06:48.216095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:48.216066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:52.293001 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:52.292965    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:06:54.604843 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:54.604755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:55.221208 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.221173    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"]
Sep 16 01:06:55.221446 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.221404    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj" podUID="f9f4380f-44f9-49ca-bafb-d035ab00a5d7" containerName="maas-api" containerID="cri-o://2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445" gracePeriod=30
Sep 16 01:06:55.276967 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.276935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:55.371286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.371255    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:55.472252 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.472180    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:55.475768 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.475737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:55.576811 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.576778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:55.611817 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.611785    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-openshift-service-ca\") pod \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") "
Sep 16 01:06:55.612297 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.611830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-maas-api-tls\") pod \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") "
Sep 16 01:06:55.612297 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.611899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-metrics-certs\") pod \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") "
Sep 16 01:06:55.612297 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.611924    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhngw\" (UniqueName: \"kubernetes.io/projected/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-kube-api-access-jhngw\") pod \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\" (UID: \"f9f4380f-44f9-49ca-bafb-d035ab00a5d7\") "
Sep 16 01:06:55.612297 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.612263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9f4380f-44f9-49ca-bafb-d035ab00a5d7" (UID: "f9f4380f-44f9-49ca-bafb-d035ab00a5d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:06:55.614545 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.614514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-kube-api-access-jhngw" (OuterVolumeSpecName: "kube-api-access-jhngw") pod "f9f4380f-44f9-49ca-bafb-d035ab00a5d7" (UID: "f9f4380f-44f9-49ca-bafb-d035ab00a5d7"). InnerVolumeSpecName "kube-api-access-jhngw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:55.614701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.614638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9f4380f-44f9-49ca-bafb-d035ab00a5d7" (UID: "f9f4380f-44f9-49ca-bafb-d035ab00a5d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:06:55.614701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.614684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f9f4380f-44f9-49ca-bafb-d035ab00a5d7" (UID: "f9f4380f-44f9-49ca-bafb-d035ab00a5d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:06:55.713195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.713159    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:55.713195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.713193    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:55.713402 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.713206    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:55.713402 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.713219    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhngw\" (UniqueName: \"kubernetes.io/projected/f9f4380f-44f9-49ca-bafb-d035ab00a5d7-kube-api-access-jhngw\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:06:55.866908 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.866870    2581 generic.go:356] "Generic (PLEG): container finished" podID="f9f4380f-44f9-49ca-bafb-d035ab00a5d7" containerID="2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445" exitCode=0
Sep 16 01:06:55.867092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.866914    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"
Sep 16 01:06:55.867092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.866958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj" event={"ID":"f9f4380f-44f9-49ca-bafb-d035ab00a5d7","Type":"ContainerDied","Data":"2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445"}
Sep 16 01:06:55.867092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.867014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj" event={"ID":"f9f4380f-44f9-49ca-bafb-d035ab00a5d7","Type":"ContainerDied","Data":"132cd006b137b854229ea01c0b08d1daefc2d794ffea88f12619c9d6912eb186"}
Sep 16 01:06:55.867092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.867041    2581 scope.go:117] "RemoveContainer" containerID="2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445"
Sep 16 01:06:55.877323 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.877301    2581 scope.go:117] "RemoveContainer" containerID="2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445"
Sep 16 01:06:55.877659 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:06:55.877635    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445\": container with ID starting with 2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445 not found: ID does not exist" containerID="2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445"
Sep 16 01:06:55.877725 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.877671    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445"} err="failed to get container status \"2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445\": rpc error: code = NotFound desc = could not find container \"2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445\": container with ID starting with 2cc0613ca3cfb312a00a9be9a20faad88518c50a6ab88904ba950fe41c472445 not found: ID does not exist"
Sep 16 01:06:55.888523 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.888476    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"]
Sep 16 01:06:55.892676 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.892649    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30fb702c-79f85656c-cp6dj"]
Sep 16 01:06:55.942051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:55.942016    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9f4380f-44f9-49ca-bafb-d035ab00a5d7" path="/var/lib/kubelet/pods/f9f4380f-44f9-49ca-bafb-d035ab00a5d7/volumes"
Sep 16 01:06:56.032423 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:56.032390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:56.673905 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:56.673868    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"]
Sep 16 01:06:56.674415 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:56.674087    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy" containerID="cri-o://505c24ad4bdd8bd01711fd3ebad29e64265fce2f7f08db6af79a8b6c4ce0a5c0" gracePeriod=30
Sep 16 01:06:56.699657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:56.699620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:56.840976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:56.840938    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:57.188755 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.188709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:57.496667 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.496579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"]
Sep 16 01:06:57.497090 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.497073    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f4380f-44f9-49ca-bafb-d035ab00a5d7" containerName="maas-api"
Sep 16 01:06:57.497138 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.497093    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f4380f-44f9-49ca-bafb-d035ab00a5d7" containerName="maas-api"
Sep 16 01:06:57.497177 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.497171    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f4380f-44f9-49ca-bafb-d035ab00a5d7" containerName="maas-api"
Sep 16 01:06:57.500217 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.500188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.502791 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.502755    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6f50fc84-openshift-default-dockercfg-27jzn\""
Sep 16 01:06:57.513674 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.513644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"]
Sep 16 01:06:57.527629 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.527591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:57.633054 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-podinfo\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-credential-socket\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e008b76-8365-456b-a0b3-a08e7d073f7e-istiod-ca-cert\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-socket\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-envoy\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-data\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-certs\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633593 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4dg\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-kube-api-access-7w4dg\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.633593 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.633377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-token\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.673802 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.673753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:57.733977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.733927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e008b76-8365-456b-a0b3-a08e7d073f7e-istiod-ca-cert\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.733977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.733985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-socket\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-envoy\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-data\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-certs\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4dg\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-kube-api-access-7w4dg\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-token\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-podinfo\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-credential-socket\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734877 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-certs\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734877 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e008b76-8365-456b-a0b3-a08e7d073f7e-istiod-ca-cert\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734877 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-data\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.734994 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.734950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-credential-socket\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.735088 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.735072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-socket\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.736571 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.736544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-envoy\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.737074 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.737051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-podinfo\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.742293 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.742266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-token\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.742547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.742510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4dg\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-kube-api-access-7w4dg\") pod \"e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") " pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:57.777506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.777444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:57.813470 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:57.813430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:58.088976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.088945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"]
Sep 16 01:06:58.091166 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:06:58.091132    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e008b76_8365_456b_a0b3_a08e7d073f7e.slice/crio-bdb5ddea271996049ea12cd19ab5c66de8f995e02a17c8d988fe369bebe8ece7 WatchSource:0}: Error finding container bdb5ddea271996049ea12cd19ab5c66de8f995e02a17c8d988fe369bebe8ece7: Status 404 returned error can't find the container with id bdb5ddea271996049ea12cd19ab5c66de8f995e02a17c8d988fe369bebe8ece7
Sep 16 01:06:58.093873 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.093841    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:58.093983 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.093919    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:58.093983 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.093965    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:06:58.512074 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.512048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:58.886759 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.886667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5" event={"ID":"0e008b76-8365-456b-a0b3-a08e7d073f7e","Type":"ContainerStarted","Data":"78ded42f318df1930c6deca06d6589d4eefb3cff55e1706b27d017faadf7f6e5"}
Sep 16 01:06:58.886759 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.886734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5" event={"ID":"0e008b76-8365-456b-a0b3-a08e7d073f7e","Type":"ContainerStarted","Data":"bdb5ddea271996049ea12cd19ab5c66de8f995e02a17c8d988fe369bebe8ece7"}
Sep 16 01:06:58.905854 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:58.905802    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5" podStartSLOduration=1.905784096 podStartE2EDuration="1.905784096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:58.90404924 +0000 UTC m=+1437.447186228" watchObservedRunningTime="2026-09-16 01:06:58.905784096 +0000 UTC m=+1437.448921072"
Sep 16 01:06:59.505072 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:59.505031    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:06:59.814407 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:59.814370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:59.819748 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:59.819720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:59.892128 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:59.892094    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:06:59.893132 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:06:59.893114    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:07:01.156580 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:01.156475    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:01.256902 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:01.256870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:01.844711 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:01.844672    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:01.904619 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:01.904587    2581 generic.go:356] "Generic (PLEG): container finished" podID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerID="505c24ad4bdd8bd01711fd3ebad29e64265fce2f7f08db6af79a8b6c4ce0a5c0" exitCode=0
Sep 16 01:07:01.904780 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:01.904656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" event={"ID":"65c235c2-95e0-4afa-8db3-051f2e1ebf8b","Type":"ContainerDied","Data":"505c24ad4bdd8bd01711fd3ebad29e64265fce2f7f08db6af79a8b6c4ce0a5c0"}
Sep 16 01:07:01.904780 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:01.904741    2581 scope.go:117] "RemoveContainer" containerID="f68f6bb4798a91a147988a8cde482c9a4189b80706d0edbd4fb10f0f4ace233a"
Sep 16 01:07:01.948071 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:01.948015    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:07:02.075606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075570    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-podinfo\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.075784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075652    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-credential-socket\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.075784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-token\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.075784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-envoy\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.075784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-data\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.075784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075785    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4zsf\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-kube-api-access-l4zsf\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.076051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075810    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-certs\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.076051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075836    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istiod-ca-cert\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.076051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.075859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-socket\") pod \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\" (UID: \"65c235c2-95e0-4afa-8db3-051f2e1ebf8b\") "
Sep 16 01:07:02.076051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:02.076327 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:02.076422 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076392    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:02.076641 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-data" (OuterVolumeSpecName: "istio-data") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:02.076742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076701    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:02.076836 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076817    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.076924 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076840    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.076924 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076855    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.076924 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076868    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.076924 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.076880    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.078139 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.078110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:02.078360 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.078328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-kube-api-access-l4zsf" (OuterVolumeSpecName: "kube-api-access-l4zsf") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "kube-api-access-l4zsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:02.078443 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.078407    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:07:02.078443 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.078411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "65c235c2-95e0-4afa-8db3-051f2e1ebf8b" (UID: "65c235c2-95e0-4afa-8db3-051f2e1ebf8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:02.177838 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.177800    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.177838 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.177833    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.177838 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.177842    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.178309 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.177850    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zsf\" (UniqueName: \"kubernetes.io/projected/65c235c2-95e0-4afa-8db3-051f2e1ebf8b-kube-api-access-l4zsf\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:02.696873 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.696836    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"]
Sep 16 01:07:02.697543 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.697475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy"
Sep 16 01:07:02.697543 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.697518    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy"
Sep 16 01:07:02.697543 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.697536    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy"
Sep 16 01:07:02.697676 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.697545    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy"
Sep 16 01:07:02.697676 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.697667    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy"
Sep 16 01:07:02.702813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.702786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.705911 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.705881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6f50fc84-serving-cert\""
Sep 16 01:07:02.707611 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.707586    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"]
Sep 16 01:07:02.737236 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.737197    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:02.797448 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.797412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:02.884883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.884847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-metrics-certs\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.885055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.884900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-maas-api-tls\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.885055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.884960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkdq\" (UniqueName: \"kubernetes.io/projected/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-kube-api-access-txkdq\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.885055 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.885010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-openshift-service-ca\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.913185 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.913152    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"
Sep 16 01:07:02.913185 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.913165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst" event={"ID":"65c235c2-95e0-4afa-8db3-051f2e1ebf8b","Type":"ContainerDied","Data":"55a210f4a54eb4e848f37e1ee5b90a279fdabc3057b4734fb41651f7aca3327e"}
Sep 16 01:07:02.913429 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.913227    2581 scope.go:117] "RemoveContainer" containerID="505c24ad4bdd8bd01711fd3ebad29e64265fce2f7f08db6af79a8b6c4ce0a5c0"
Sep 16 01:07:02.938176 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.938142    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"]
Sep 16 01:07:02.946277 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.946242    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-30fb702c-openshift-default-c4f568d6b-24wst"]
Sep 16 01:07:02.985789 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.985706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkdq\" (UniqueName: \"kubernetes.io/projected/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-kube-api-access-txkdq\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.985789 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.985753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-openshift-service-ca\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.986019 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.985820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-metrics-certs\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.986019 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.985871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-maas-api-tls\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.986640 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.986616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-openshift-service-ca\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.988603 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.988581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-metrics-certs\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.988696 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.988579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-maas-api-tls\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:02.993023 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:02.992992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkdq\" (UniqueName: \"kubernetes.io/projected/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-kube-api-access-txkdq\") pod \"maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:03.017751 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.017717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:03.021042 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.021011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:03.206027 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.205985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:03.292184 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.292158    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"]
Sep 16 01:07:03.293456 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.293426    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:03.294111 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:07:03.294088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2134d04_0177_4aed_aa2b_8e8de7e2a18c.slice/crio-40e8d63a5166a12a6e0ac0e7182074993bef59fce6e56fe0167b0a96c03af895 WatchSource:0}: Error finding container 40e8d63a5166a12a6e0ac0e7182074993bef59fce6e56fe0167b0a96c03af895: Status 404 returned error can't find the container with id 40e8d63a5166a12a6e0ac0e7182074993bef59fce6e56fe0167b0a96c03af895
Sep 16 01:07:03.602286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.602247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:03.923231 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.923128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf" event={"ID":"a2134d04-0177-4aed-aa2b-8e8de7e2a18c","Type":"ContainerStarted","Data":"430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2"}
Sep 16 01:07:03.923231 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.923194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:03.923231 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.923218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf" event={"ID":"a2134d04-0177-4aed-aa2b-8e8de7e2a18c","Type":"ContainerStarted","Data":"40e8d63a5166a12a6e0ac0e7182074993bef59fce6e56fe0167b0a96c03af895"}
Sep 16 01:07:03.942270 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.942226    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" path="/var/lib/kubelet/pods/65c235c2-95e0-4afa-8db3-051f2e1ebf8b/volumes"
Sep 16 01:07:03.943440 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:03.943395    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf" podStartSLOduration=1.6400573010000001 podStartE2EDuration="1.943380356s" totalImagesPullingTime="303.323055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:07:03.941684436 +0000 UTC m=+1442.484821411" watchObservedRunningTime="2026-09-16 01:07:03.943380356 +0000 UTC m=+1442.486517330"
Sep 16 01:07:04.419941 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:04.419854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:05.087033 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:05.086990    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:09.941547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:09.941514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:24.485936 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:24.485891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:07:24.826173 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:24.826133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:25.533555 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:25.533522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:26.345124 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:26.345088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:27.115615 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:27.115581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:28.023086 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:28.023053    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:28.898436 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:28.898403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:30.617551 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:30.617501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:30.921677 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:30.921584    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:31.645749 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.645717    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"]
Sep 16 01:07:31.646182 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.645927    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf" podUID="a2134d04-0177-4aed-aa2b-8e8de7e2a18c" containerName="maas-api" containerID="cri-o://430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2" gracePeriod=30
Sep 16 01:07:31.679081 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.678899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:31.746697 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.746662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:31.808419 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.808387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:31.876629 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.876359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:31.932503 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.932444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:31.933500 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:31.933471    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:32.024222 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.024183    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"]
Sep 16 01:07:32.024498 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.024456    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5" podUID="0e008b76-8365-456b-a0b3-a08e7d073f7e" containerName="istio-proxy" containerID="cri-o://78ded42f318df1930c6deca06d6589d4eefb3cff55e1706b27d017faadf7f6e5" gracePeriod=30
Sep 16 01:07:32.038522 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.038493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:32.084855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.083844    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-openshift-service-ca\") pod \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") "
Sep 16 01:07:32.084855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.083908    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-metrics-certs\") pod \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") "
Sep 16 01:07:32.084855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.083988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-maas-api-tls\") pod \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") "
Sep 16 01:07:32.084855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.084021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txkdq\" (UniqueName: \"kubernetes.io/projected/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-kube-api-access-txkdq\") pod \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\" (UID: \"a2134d04-0177-4aed-aa2b-8e8de7e2a18c\") "
Sep 16 01:07:32.085689 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.085652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2134d04-0177-4aed-aa2b-8e8de7e2a18c" (UID: "a2134d04-0177-4aed-aa2b-8e8de7e2a18c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:32.087072 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.087044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-kube-api-access-txkdq" (OuterVolumeSpecName: "kube-api-access-txkdq") pod "a2134d04-0177-4aed-aa2b-8e8de7e2a18c" (UID: "a2134d04-0177-4aed-aa2b-8e8de7e2a18c"). InnerVolumeSpecName "kube-api-access-txkdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:32.087349 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.087322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a2134d04-0177-4aed-aa2b-8e8de7e2a18c" (UID: "a2134d04-0177-4aed-aa2b-8e8de7e2a18c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:07:32.087563 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.087538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2134d04-0177-4aed-aa2b-8e8de7e2a18c" (UID: "a2134d04-0177-4aed-aa2b-8e8de7e2a18c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:07:32.100567 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.100537    2581 generic.go:356] "Generic (PLEG): container finished" podID="a2134d04-0177-4aed-aa2b-8e8de7e2a18c" containerID="430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2" exitCode=0
Sep 16 01:07:32.100708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.100574    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"
Sep 16 01:07:32.100708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.100574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf" event={"ID":"a2134d04-0177-4aed-aa2b-8e8de7e2a18c","Type":"ContainerDied","Data":"430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2"}
Sep 16 01:07:32.100708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.100644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf" event={"ID":"a2134d04-0177-4aed-aa2b-8e8de7e2a18c","Type":"ContainerDied","Data":"40e8d63a5166a12a6e0ac0e7182074993bef59fce6e56fe0167b0a96c03af895"}
Sep 16 01:07:32.100708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.100677    2581 scope.go:117] "RemoveContainer" containerID="430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2"
Sep 16 01:07:32.111474 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.111455    2581 scope.go:117] "RemoveContainer" containerID="430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2"
Sep 16 01:07:32.111802 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:07:32.111782    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2\": container with ID starting with 430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2 not found: ID does not exist" containerID="430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2"
Sep 16 01:07:32.111864 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.111814    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2"} err="failed to get container status \"430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2\": rpc error: code = NotFound desc = could not find container \"430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2\": container with ID starting with 430bf2bfc85fd529f2fe95205b7bd6e9fe23712f00cff2f23bcf1aa08f8ffed2 not found: ID does not exist"
Sep 16 01:07:32.122019 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.121989    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"]
Sep 16 01:07:32.125878 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.125851    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f50fc84-87d497d8c-gphpf"]
Sep 16 01:07:32.185366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.185273    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:32.185366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.185315    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:32.185366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.185332    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:32.185366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.185346    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txkdq\" (UniqueName: \"kubernetes.io/projected/a2134d04-0177-4aed-aa2b-8e8de7e2a18c-kube-api-access-txkdq\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:32.730178 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.730132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:32.828453 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.828412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:32.975184 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:32.975144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:33.674119 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:33.674086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:33.942739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:33.942659    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2134d04-0177-4aed-aa2b-8e8de7e2a18c" path="/var/lib/kubelet/pods/a2134d04-0177-4aed-aa2b-8e8de7e2a18c/volumes"
Sep 16 01:07:34.370822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:34.370780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:34.515475 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:34.515441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:35.186129 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:35.186092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:35.603866 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:35.603819    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"]
Sep 16 01:07:35.698360 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:35.698321    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:36.129906 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.129875    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e008b76-8365-456b-a0b3-a08e7d073f7e" containerID="78ded42f318df1930c6deca06d6589d4eefb3cff55e1706b27d017faadf7f6e5" exitCode=0
Sep 16 01:07:36.130108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.129948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5" event={"ID":"0e008b76-8365-456b-a0b3-a08e7d073f7e","Type":"ContainerDied","Data":"78ded42f318df1930c6deca06d6589d4eefb3cff55e1706b27d017faadf7f6e5"}
Sep 16 01:07:36.185933 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.185907    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:07:36.326036 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.325991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w4dg\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-kube-api-access-7w4dg\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e008b76-8365-456b-a0b3-a08e7d073f7e-istiod-ca-cert\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-podinfo\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326155    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-socket\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-envoy\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-token\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-credential-socket\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326339    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-data\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326619    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:36.326979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326631    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e008b76-8365-456b-a0b3-a08e7d073f7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:36.326979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:36.326979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-certs\") pod \"0e008b76-8365-456b-a0b3-a08e7d073f7e\" (UID: \"0e008b76-8365-456b-a0b3-a08e7d073f7e\") "
Sep 16 01:07:36.326979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:36.326979 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.326891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:36.327312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.327122    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e008b76-8365-456b-a0b3-a08e7d073f7e-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.327312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.327141    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.327312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.327153    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.327312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.327167    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.327312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.327179    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.328591 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.328564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:36.328684 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.328640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:07:36.328684 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.328656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-kube-api-access-7w4dg" (OuterVolumeSpecName: "kube-api-access-7w4dg") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "kube-api-access-7w4dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:36.328767 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.328743    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e008b76-8365-456b-a0b3-a08e7d073f7e" (UID: "0e008b76-8365-456b-a0b3-a08e7d073f7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:36.427753 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.427700    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.427753 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.427748    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.427753 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.427757    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.427753 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.427765    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4dg\" (UniqueName: \"kubernetes.io/projected/0e008b76-8365-456b-a0b3-a08e7d073f7e-kube-api-access-7w4dg\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:36.474812 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:36.474777    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:37.136929 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:37.136897    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"
Sep 16 01:07:37.137113 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:37.136929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5" event={"ID":"0e008b76-8365-456b-a0b3-a08e7d073f7e","Type":"ContainerDied","Data":"bdb5ddea271996049ea12cd19ab5c66de8f995e02a17c8d988fe369bebe8ece7"}
Sep 16 01:07:37.137113 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:37.136991    2581 scope.go:117] "RemoveContainer" containerID="78ded42f318df1930c6deca06d6589d4eefb3cff55e1706b27d017faadf7f6e5"
Sep 16 01:07:37.158554 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:37.158526    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"]
Sep 16 01:07:37.162215 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:37.162192    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6f50fc84-openshift-default-b796d5576-9z8j5"]
Sep 16 01:07:37.948840 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:37.948801    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e008b76-8365-456b-a0b3-a08e7d073f7e" path="/var/lib/kubelet/pods/0e008b76-8365-456b-a0b3-a08e7d073f7e/volumes"
Sep 16 01:07:39.267744 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.267710    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"]
Sep 16 01:07:39.268126 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.268112    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2134d04-0177-4aed-aa2b-8e8de7e2a18c" containerName="maas-api"
Sep 16 01:07:39.268170 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.268128    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2134d04-0177-4aed-aa2b-8e8de7e2a18c" containerName="maas-api"
Sep 16 01:07:39.268170 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.268142    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e008b76-8365-456b-a0b3-a08e7d073f7e" containerName="istio-proxy"
Sep 16 01:07:39.268170 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.268147    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e008b76-8365-456b-a0b3-a08e7d073f7e" containerName="istio-proxy"
Sep 16 01:07:39.268310 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.268220    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2134d04-0177-4aed-aa2b-8e8de7e2a18c" containerName="maas-api"
Sep 16 01:07:39.268310 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.268229    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e008b76-8365-456b-a0b3-a08e7d073f7e" containerName="istio-proxy"
Sep 16 01:07:39.268310 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.268236    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c235c2-95e0-4afa-8db3-051f2e1ebf8b" containerName="istio-proxy"
Sep 16 01:07:39.271428 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.271409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.273698 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.273674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f831c5\"/\"auth-test-model-f831c5-kserve-self-signed-certs\""
Sep 16 01:07:39.274658 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.274637    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f831c5\"/\"kube-root-ca.crt\""
Sep 16 01:07:39.274758 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.274707    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f831c5\"/\"openshift-service-ca.crt\""
Sep 16 01:07:39.274758 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.274707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f831c5\"/\"default-dockercfg-kqqjd\""
Sep 16 01:07:39.280886 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.280863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"]
Sep 16 01:07:39.360469 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.360428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-model-cache\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.360688 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.360477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-dshm\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.360688 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.360530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tls-certs\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.360688 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.360562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-home\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.360688 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.360661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tmp-dir\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.360886 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.360732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824gr\" (UniqueName: \"kubernetes.io/projected/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-kube-api-access-824gr\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.461696 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.461652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tmp-dir\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.461876 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.461730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824gr\" (UniqueName: \"kubernetes.io/projected/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-kube-api-access-824gr\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.461876 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.461767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-model-cache\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.461965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.461902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-dshm\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.461965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.461953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tls-certs\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.462041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.462013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-home\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.462157 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.462136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tmp-dir\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.462246 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.462144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-model-cache\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.462368 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.462348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-home\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.464169 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.464137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-dshm\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.464420 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.464400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tls-certs\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.469425 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.469404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824gr\" (UniqueName: \"kubernetes.io/projected/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-kube-api-access-824gr\") pod \"auth-test-model-f831c5-kserve-ccf569b59-hzkl5\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.577401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.577319    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:39.582287 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.582267    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:39.852647 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.852617    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"]
Sep 16 01:07:39.854984 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:07:39.854958    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1d22e3_bb0a_4a02_a52a_59ca07415e39.slice/crio-3fe312ec6cfae0784f1b86f8c6b448df543c5689cb732dc91af6e897e3eeadd1 WatchSource:0}: Error finding container 3fe312ec6cfae0784f1b86f8c6b448df543c5689cb732dc91af6e897e3eeadd1: Status 404 returned error can't find the container with id 3fe312ec6cfae0784f1b86f8c6b448df543c5689cb732dc91af6e897e3eeadd1
Sep 16 01:07:39.996790 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:39.996753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:40.104032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.103937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:40.157359 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.157320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5" event={"ID":"3d1d22e3-bb0a-4a02-a52a-59ca07415e39","Type":"ContainerStarted","Data":"8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633"}
Sep 16 01:07:40.157561 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.157369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5" event={"ID":"3d1d22e3-bb0a-4a02-a52a-59ca07415e39","Type":"ContainerStarted","Data":"3fe312ec6cfae0784f1b86f8c6b448df543c5689cb732dc91af6e897e3eeadd1"}
Sep 16 01:07:40.178851 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.178799    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5" podStartSLOduration=1.178782253 podStartE2EDuration="1.178782253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:07:40.177432241 +0000 UTC m=+1478.720569219" watchObservedRunningTime="2026-09-16 01:07:40.178782253 +0000 UTC m=+1478.721919240"
Sep 16 01:07:40.197680 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.197642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:40.297376 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.297339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:40.763647 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.763612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:40.929194 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:40.929159    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:46.597126 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.597090    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"]
Sep 16 01:07:46.597629 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.597302    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s" podUID="64fcd223-dd24-4ecf-992c-70b1df7b335e" containerName="maas-api" containerID="cri-o://4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704" gracePeriod=30
Sep 16 01:07:46.621903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.621871    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:46.688260 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.688231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:46.789183 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.789149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:46.852162 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.852089    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:07:46.871775 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.871744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:46.932299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.932244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwvt6\" (UniqueName: \"kubernetes.io/projected/64fcd223-dd24-4ecf-992c-70b1df7b335e-kube-api-access-dwvt6\") pod \"64fcd223-dd24-4ecf-992c-70b1df7b335e\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") "
Sep 16 01:07:46.932299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.932312    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-maas-api-tls\") pod \"64fcd223-dd24-4ecf-992c-70b1df7b335e\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") "
Sep 16 01:07:46.932580 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.932337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fcd223-dd24-4ecf-992c-70b1df7b335e-openshift-service-ca\") pod \"64fcd223-dd24-4ecf-992c-70b1df7b335e\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") "
Sep 16 01:07:46.932580 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.932383    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-metrics-certs\") pod \"64fcd223-dd24-4ecf-992c-70b1df7b335e\" (UID: \"64fcd223-dd24-4ecf-992c-70b1df7b335e\") "
Sep 16 01:07:46.932795 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.932766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fcd223-dd24-4ecf-992c-70b1df7b335e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64fcd223-dd24-4ecf-992c-70b1df7b335e" (UID: "64fcd223-dd24-4ecf-992c-70b1df7b335e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:46.934930 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.934886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "64fcd223-dd24-4ecf-992c-70b1df7b335e" (UID: "64fcd223-dd24-4ecf-992c-70b1df7b335e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:07:46.935129 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.935102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64fcd223-dd24-4ecf-992c-70b1df7b335e" (UID: "64fcd223-dd24-4ecf-992c-70b1df7b335e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:07:46.935129 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:46.935123    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fcd223-dd24-4ecf-992c-70b1df7b335e-kube-api-access-dwvt6" (OuterVolumeSpecName: "kube-api-access-dwvt6") pod "64fcd223-dd24-4ecf-992c-70b1df7b335e" (UID: "64fcd223-dd24-4ecf-992c-70b1df7b335e"). InnerVolumeSpecName "kube-api-access-dwvt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:47.033570 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.033530    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwvt6\" (UniqueName: \"kubernetes.io/projected/64fcd223-dd24-4ecf-992c-70b1df7b335e-kube-api-access-dwvt6\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:47.033570 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.033567    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:47.033570 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.033577    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fcd223-dd24-4ecf-992c-70b1df7b335e-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:47.033825 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.033586    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fcd223-dd24-4ecf-992c-70b1df7b335e-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:47.196535 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.196437    2581 generic.go:356] "Generic (PLEG): container finished" podID="64fcd223-dd24-4ecf-992c-70b1df7b335e" containerID="4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704" exitCode=0
Sep 16 01:07:47.196535 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.196495    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"
Sep 16 01:07:47.196535 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.196521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s" event={"ID":"64fcd223-dd24-4ecf-992c-70b1df7b335e","Type":"ContainerDied","Data":"4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704"}
Sep 16 01:07:47.196758 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.196566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s" event={"ID":"64fcd223-dd24-4ecf-992c-70b1df7b335e","Type":"ContainerDied","Data":"a486b1376ea9d7ffce381e1a38b1fd853aaff88fdc6bd524bd53885c2313f651"}
Sep 16 01:07:47.196758 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.196592    2581 scope.go:117] "RemoveContainer" containerID="4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704"
Sep 16 01:07:47.206663 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.206642    2581 scope.go:117] "RemoveContainer" containerID="4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704"
Sep 16 01:07:47.207039 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:07:47.207019    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704\": container with ID starting with 4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704 not found: ID does not exist" containerID="4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704"
Sep 16 01:07:47.207105 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.207046    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704"} err="failed to get container status \"4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704\": rpc error: code = NotFound desc = could not find container \"4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704\": container with ID starting with 4ef5b226eb56bf2d05c47a28d8c6b3bbe5afea8c0427c2f8fbcdc02bb0c32704 not found: ID does not exist"
Sep 16 01:07:47.221345 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.221314    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"]
Sep 16 01:07:47.225281 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.225256    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2534e3-7f84fb9669-jnw7s"]
Sep 16 01:07:47.230644 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.230613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:47.350476 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.350436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:47.941844 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:47.941804    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fcd223-dd24-4ecf-992c-70b1df7b335e" path="/var/lib/kubelet/pods/64fcd223-dd24-4ecf-992c-70b1df7b335e/volumes"
Sep 16 01:07:48.026707 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:48.026673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:48.828510 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:48.828464    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:49.517830 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:49.517797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:49.582981 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:49.582948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:49.583150 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:49.583028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:49.596253 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:49.596225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:49.615371 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:49.615339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:50.228164 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.228135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:07:50.624226 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.624161    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq" podUID="ee1251d1-7d4d-410e-8010-a10bcf15691e" containerName="main" containerID="cri-o://9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d" gracePeriod=45
Sep 16 01:07:50.878893 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.878825    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:07:50.971862 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.971828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1251d1-7d4d-410e-8010-a10bcf15691e-tls-certs\") pod \"ee1251d1-7d4d-410e-8010-a10bcf15691e\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") "
Sep 16 01:07:50.972077 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.971909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stnw4\" (UniqueName: \"kubernetes.io/projected/ee1251d1-7d4d-410e-8010-a10bcf15691e-kube-api-access-stnw4\") pod \"ee1251d1-7d4d-410e-8010-a10bcf15691e\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") "
Sep 16 01:07:50.972077 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.971955    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-dshm\") pod \"ee1251d1-7d4d-410e-8010-a10bcf15691e\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") "
Sep 16 01:07:50.972077 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.971988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-model-cache\") pod \"ee1251d1-7d4d-410e-8010-a10bcf15691e\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") "
Sep 16 01:07:50.972077 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.972038    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-tmp-dir\") pod \"ee1251d1-7d4d-410e-8010-a10bcf15691e\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") "
Sep 16 01:07:50.972308 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.972091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-home\") pod \"ee1251d1-7d4d-410e-8010-a10bcf15691e\" (UID: \"ee1251d1-7d4d-410e-8010-a10bcf15691e\") "
Sep 16 01:07:50.972391 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.972352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee1251d1-7d4d-410e-8010-a10bcf15691e" (UID: "ee1251d1-7d4d-410e-8010-a10bcf15691e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:50.972460 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.972392    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-home" (OuterVolumeSpecName: "home") pod "ee1251d1-7d4d-410e-8010-a10bcf15691e" (UID: "ee1251d1-7d4d-410e-8010-a10bcf15691e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:50.972797 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.972602    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-tmp-dir\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:50.972797 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.972627    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-home\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:50.972797 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.972738    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee1251d1-7d4d-410e-8010-a10bcf15691e" (UID: "ee1251d1-7d4d-410e-8010-a10bcf15691e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:50.974967 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.974939    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1251d1-7d4d-410e-8010-a10bcf15691e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee1251d1-7d4d-410e-8010-a10bcf15691e" (UID: "ee1251d1-7d4d-410e-8010-a10bcf15691e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:07:50.975206 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.975188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-dshm" (OuterVolumeSpecName: "dshm") pod "ee1251d1-7d4d-410e-8010-a10bcf15691e" (UID: "ee1251d1-7d4d-410e-8010-a10bcf15691e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:50.975283 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:50.975267    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1251d1-7d4d-410e-8010-a10bcf15691e-kube-api-access-stnw4" (OuterVolumeSpecName: "kube-api-access-stnw4") pod "ee1251d1-7d4d-410e-8010-a10bcf15691e" (UID: "ee1251d1-7d4d-410e-8010-a10bcf15691e"). InnerVolumeSpecName "kube-api-access-stnw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:51.073963 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.073924    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1251d1-7d4d-410e-8010-a10bcf15691e-tls-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.073963 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.073961    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnw4\" (UniqueName: \"kubernetes.io/projected/ee1251d1-7d4d-410e-8010-a10bcf15691e-kube-api-access-stnw4\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.073963 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.073978    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-dshm\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.074392 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.073990    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1251d1-7d4d-410e-8010-a10bcf15691e-model-cache\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.222934 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.222837    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee1251d1-7d4d-410e-8010-a10bcf15691e" containerID="9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d" exitCode=0
Sep 16 01:07:51.222934 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.222885    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"
Sep 16 01:07:51.222934 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.222927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq" event={"ID":"ee1251d1-7d4d-410e-8010-a10bcf15691e","Type":"ContainerDied","Data":"9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d"}
Sep 16 01:07:51.223229 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.222974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq" event={"ID":"ee1251d1-7d4d-410e-8010-a10bcf15691e","Type":"ContainerDied","Data":"f22cd5f032a538a22f92b836aae2218c8fbf2c16294f4a4c41312a19d118c2f1"}
Sep 16 01:07:51.223229 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.222998    2581 scope.go:117] "RemoveContainer" containerID="9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d"
Sep 16 01:07:51.232879 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.232862    2581 scope.go:117] "RemoveContainer" containerID="9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d"
Sep 16 01:07:51.233156 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:07:51.233139    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d\": container with ID starting with 9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d not found: ID does not exist" containerID="9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d"
Sep 16 01:07:51.233208 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.233163    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d"} err="failed to get container status \"9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d\": rpc error: code = NotFound desc = could not find container \"9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d\": container with ID starting with 9f5777314f245f493b05782e31c304ee0e0a5803e3be0ce47d53cc6a60e7546d not found: ID does not exist"
Sep 16 01:07:51.243722 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.243698    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"]
Sep 16 01:07:51.247085 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.247060    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-2-w3-2534e3-kserve-77b65f965-qzgkq"]
Sep 16 01:07:51.941014 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:51.940978    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1251d1-7d4d-410e-8010-a10bcf15691e" path="/var/lib/kubelet/pods/ee1251d1-7d4d-410e-8010-a10bcf15691e/volumes"
Sep 16 01:07:52.723160 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:52.723103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:52.837253 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:52.837217    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:53.032835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:53.032798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:07:59.587723 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:07:59.587683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:08:01.975060 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:01.975029    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:08:01.980709 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:01.980682    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:08:03.513807 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:03.513764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:06.725046 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:06.725006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:12.276418 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:12.276380    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:12.338122 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:12.338084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:12.408674 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:12.408640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:12.505431 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:12.505396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:19.132244 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:19.132190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:19.232291 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:19.232249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:19.506051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:19.505955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:21.684360 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:21.684323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:08:28.358944 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:28.358832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:33.973463 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:33.973413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:34.033081 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:34.033046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:34.113669 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:34.113622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:34.214071 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:34.214023    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:34.406464 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:34.406427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:41.779642 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:41.779611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:44.668946 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:44.668889    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:45.546016 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:45.545973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:48.183397 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:48.183356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:08:51.289030 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:51.288992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:51.395077 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:51.395027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:51.488923 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:51.488886    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:51.588653 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:51.588548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:52.227061 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:52.227020    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:52.297966 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:52.297929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:52.397954 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:52.397908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:08:54.015615 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:08:54.015581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:00.008875 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:00.008816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:00.068412 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:00.068361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:00.708206 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:00.708175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:06.378000 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.377963    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:06.439265 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.439230    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:06.628750 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.628651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:06.907068 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.906969    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"]
Sep 16 01:09:06.907794 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.907768    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1251d1-7d4d-410e-8010-a10bcf15691e" containerName="main"
Sep 16 01:09:06.907794 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.907795    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1251d1-7d4d-410e-8010-a10bcf15691e" containerName="main"
Sep 16 01:09:06.907945 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.907813    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fcd223-dd24-4ecf-992c-70b1df7b335e" containerName="maas-api"
Sep 16 01:09:06.907945 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.907822    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fcd223-dd24-4ecf-992c-70b1df7b335e" containerName="maas-api"
Sep 16 01:09:06.907945 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.907908    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1251d1-7d4d-410e-8010-a10bcf15691e" containerName="main"
Sep 16 01:09:06.907945 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.907926    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fcd223-dd24-4ecf-992c-70b1df7b335e" containerName="maas-api"
Sep 16 01:09:06.911232 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.911214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:06.913620 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.913600    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 01:09:06.913722 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.913623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f29ccae3-openshift-default-dockercfg-t8tnt\""
Sep 16 01:09:06.922706 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.922678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"]
Sep 16 01:09:06.930628 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:06.930595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:07.091738 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.091704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-certs\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.091953 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.091777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-credential-socket\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.091953 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.091838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-podinfo\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.091953 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.091915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istiod-ca-cert\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.092157 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.092122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-envoy\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.092222 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.092188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-data\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.092275 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.092228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-token\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.092473 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.092439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-socket\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.092628 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.092577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcm5\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-kube-api-access-bkcm5\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.125015 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.124980    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:07.193290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-credential-socket\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-podinfo\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istiod-ca-cert\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-envoy\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-data\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-token\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-socket\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcm5\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-kube-api-access-bkcm5\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-certs\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-credential-socket\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.193882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-data\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.194056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-socket\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.194056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.193990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-certs\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.194210 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.194189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istiod-ca-cert\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.195612 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.195596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-envoy\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.195855 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.195836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-podinfo\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.200475 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.200450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-token\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.200824 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.200806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcm5\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-kube-api-access-bkcm5\") pod \"e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") " pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.225900 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.225851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:07.504350 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.504325    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"]
Sep 16 01:09:07.506948 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:09:07.506922    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36c2b28_4782_4560_bfb2_5e82fe2c2c44.slice/crio-d3ebfec69bb046a7138ad1b50256abd495a7df2c05744694ffc329b4f785654c WatchSource:0}: Error finding container d3ebfec69bb046a7138ad1b50256abd495a7df2c05744694ffc329b4f785654c: Status 404 returned error can't find the container with id d3ebfec69bb046a7138ad1b50256abd495a7df2c05744694ffc329b4f785654c
Sep 16 01:09:07.509136 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.509107    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:09:07.509194 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.509168    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:09:07.509237 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.509196    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:09:07.625276 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.625235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" event={"ID":"d36c2b28-4782-4560-bfb2-5e82fe2c2c44","Type":"ContainerStarted","Data":"6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2"}
Sep 16 01:09:07.625448 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.625305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" event={"ID":"d36c2b28-4782-4560-bfb2-5e82fe2c2c44","Type":"ContainerStarted","Data":"d3ebfec69bb046a7138ad1b50256abd495a7df2c05744694ffc329b4f785654c"}
Sep 16 01:09:07.644971 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:07.644910    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podStartSLOduration=1.644893552 podStartE2EDuration="1.644893552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:09:07.642318763 +0000 UTC m=+1566.185455739" watchObservedRunningTime="2026-09-16 01:09:07.644893552 +0000 UTC m=+1566.188030526"
Sep 16 01:09:08.226974 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:08.226931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:08.232307 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:08.232278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:08.630712 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:08.630678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:08.631564 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:08.631545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:09.435564 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:09.435524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:09.936599 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:09.936562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:10.027724 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:10.027680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:10.323547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:10.323509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:11.070315 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:11.070271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:12.005937 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.005902    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"]
Sep 16 01:09:12.010172 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.010147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.012547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.012363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f29ccae3-serving-cert\""
Sep 16 01:09:12.019569 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.019536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"]
Sep 16 01:09:12.036222 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.036194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:12.044523 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.044467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-metrics-certs\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.044651 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.044535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c9v7\" (UniqueName: \"kubernetes.io/projected/f36e7ea4-4566-40e0-b29e-7fe767bb895b-kube-api-access-4c9v7\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.044651 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.044579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-maas-api-tls\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.044651 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.044613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7ea4-4566-40e0-b29e-7fe767bb895b-openshift-service-ca\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.136550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.136506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:12.145757 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.145732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-metrics-certs\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.145853 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.145767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c9v7\" (UniqueName: \"kubernetes.io/projected/f36e7ea4-4566-40e0-b29e-7fe767bb895b-kube-api-access-4c9v7\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.145853 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.145815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-maas-api-tls\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.145853 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.145846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7ea4-4566-40e0-b29e-7fe767bb895b-openshift-service-ca\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.146562 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.146540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7ea4-4566-40e0-b29e-7fe767bb895b-openshift-service-ca\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.148138 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.148118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-maas-api-tls\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.148221 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.148185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-metrics-certs\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.152697 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.152676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c9v7\" (UniqueName: \"kubernetes.io/projected/f36e7ea4-4566-40e0-b29e-7fe767bb895b-kube-api-access-4c9v7\") pod \"maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.237342 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.237311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:12.326897 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.326865    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:12.501911 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.501873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:12.602594 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.602563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"]
Sep 16 01:09:12.604739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.604711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:12.605131 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:09:12.605110    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36e7ea4_4566_40e0_b29e_7fe767bb895b.slice/crio-7fbd239c4ea3730acf00ef149a932fea60868b7bbb592965c482158dc8468dba WatchSource:0}: Error finding container 7fbd239c4ea3730acf00ef149a932fea60868b7bbb592965c482158dc8468dba: Status 404 returned error can't find the container with id 7fbd239c4ea3730acf00ef149a932fea60868b7bbb592965c482158dc8468dba
Sep 16 01:09:12.655567 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.655521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd" event={"ID":"f36e7ea4-4566-40e0-b29e-7fe767bb895b","Type":"ContainerStarted","Data":"7fbd239c4ea3730acf00ef149a932fea60868b7bbb592965c482158dc8468dba"}
Sep 16 01:09:12.911033 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:12.911006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:13.661678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:13.661646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd" event={"ID":"f36e7ea4-4566-40e0-b29e-7fe767bb895b","Type":"ContainerStarted","Data":"c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0"}
Sep 16 01:09:13.662081 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:13.661719    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:13.680328 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:13.680278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd" podStartSLOduration=2.377441358 podStartE2EDuration="2.680262085s" totalImagesPullingTime="302.820727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:09:13.678612886 +0000 UTC m=+1572.221749860" watchObservedRunningTime="2026-09-16 01:09:13.680262085 +0000 UTC m=+1572.223399064"
Sep 16 01:09:13.733460 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:13.733428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:15.881051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:15.881006    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:09:19.672724 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:19.672681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:34.126558 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:34.126522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:34.844396 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:34.844357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:34.943765 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:34.943725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:35.534609 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:35.534569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:35.632451 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:35.632411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:35.736696 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:35.736663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:35.985938 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:35.985833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:36.325368 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:36.325334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:37.502366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:37.502328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:37.771309 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:37.771273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:38.903339 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:38.903305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:39.575168 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:39.575115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:39.672687 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:39.672647    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 01:09:39.672848 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:39.672755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 01:09:40.185805 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.185754    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:09:40.328154 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.328122    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:40.825387 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.825356    2581 generic.go:356] "Generic (PLEG): container finished" podID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerID="6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2" exitCode=0
Sep 16 01:09:40.825619 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.825431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" event={"ID":"d36c2b28-4782-4560-bfb2-5e82fe2c2c44","Type":"ContainerDied","Data":"6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2"}
Sep 16 01:09:40.825922 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.825908    2581 scope.go:117] "RemoveContainer" containerID="6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2"
Sep 16 01:09:40.827091 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.827061    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:09:40.827190 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.827138    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:09:40.827190 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:40.827172    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:09:41.134151 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:41.134065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:41.834947 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:41.834916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" event={"ID":"d36c2b28-4782-4560-bfb2-5e82fe2c2c44","Type":"ContainerStarted","Data":"9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755"}
Sep 16 01:09:42.162520 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.162385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:42.219745 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.219710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:42.226674 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.226654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:42.226877 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.226856    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 01:09:42.226938 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.226915    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 01:09:42.520892 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.520853    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"]
Sep 16 01:09:42.588610 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.588573    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:42.876862 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:42.876770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:43.227059 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:43.226958    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 01:09:43.227059 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:43.227040    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 01:09:43.915882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:43.915846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:44.227272 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:44.227165    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 01:09:44.227272 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:44.227239    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 01:09:44.588585 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:44.588542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:45.226392 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:45.226350    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 01:09:45.226843 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:45.226412    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 01:09:45.412977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:45.412945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:46.094412 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:46.094375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:46.227172 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:46.227137    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 01:09:46.227579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:46.227198    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 01:09:47.226274 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:47.226233    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 01:09:47.226577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:47.226305    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 01:09:47.911528 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:47.911465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:48.229925 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:48.229849    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:48.230145 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:48.230124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:48.230760 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:48.230744    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:09:52.673702 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:52.673654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.221680 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.221645    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"]
Sep 16 01:09:53.221955 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.221922    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd" podUID="f36e7ea4-4566-40e0-b29e-7fe767bb895b" containerName="maas-api" containerID="cri-o://c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0" gracePeriod=30
Sep 16 01:09:53.247279 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.247219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.348230 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.348205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.447182 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.447153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.473110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.473040    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:53.533591 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.533555    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.567442 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.567407    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-metrics-certs\") pod \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") "
Sep 16 01:09:53.567649 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.567466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c9v7\" (UniqueName: \"kubernetes.io/projected/f36e7ea4-4566-40e0-b29e-7fe767bb895b-kube-api-access-4c9v7\") pod \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") "
Sep 16 01:09:53.567649 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.567515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-maas-api-tls\") pod \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") "
Sep 16 01:09:53.567782 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.567717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7ea4-4566-40e0-b29e-7fe767bb895b-openshift-service-ca\") pod \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\" (UID: \"f36e7ea4-4566-40e0-b29e-7fe767bb895b\") "
Sep 16 01:09:53.568073 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.568043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36e7ea4-4566-40e0-b29e-7fe767bb895b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f36e7ea4-4566-40e0-b29e-7fe767bb895b" (UID: "f36e7ea4-4566-40e0-b29e-7fe767bb895b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:09:53.569643 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.569615    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f36e7ea4-4566-40e0-b29e-7fe767bb895b" (UID: "f36e7ea4-4566-40e0-b29e-7fe767bb895b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:09:53.569737 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.569636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36e7ea4-4566-40e0-b29e-7fe767bb895b-kube-api-access-4c9v7" (OuterVolumeSpecName: "kube-api-access-4c9v7") pod "f36e7ea4-4566-40e0-b29e-7fe767bb895b" (UID: "f36e7ea4-4566-40e0-b29e-7fe767bb895b"). InnerVolumeSpecName "kube-api-access-4c9v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:09:53.569737 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.569636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f36e7ea4-4566-40e0-b29e-7fe767bb895b" (UID: "f36e7ea4-4566-40e0-b29e-7fe767bb895b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:09:53.588189 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.588158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.669610 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.669570    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7ea4-4566-40e0-b29e-7fe767bb895b-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:53.669610 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.669609    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:53.669871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.669624    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c9v7\" (UniqueName: \"kubernetes.io/projected/f36e7ea4-4566-40e0-b29e-7fe767bb895b-kube-api-access-4c9v7\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:53.669871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.669637    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f36e7ea4-4566-40e0-b29e-7fe767bb895b-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:53.688803 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.688767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.741101 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.741014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.836282 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.836243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.907593 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.907555    2581 generic.go:356] "Generic (PLEG): container finished" podID="f36e7ea4-4566-40e0-b29e-7fe767bb895b" containerID="c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0" exitCode=0
Sep 16 01:09:53.907785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.907602    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"
Sep 16 01:09:53.907785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.907641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd" event={"ID":"f36e7ea4-4566-40e0-b29e-7fe767bb895b","Type":"ContainerDied","Data":"c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0"}
Sep 16 01:09:53.907785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.907693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd" event={"ID":"f36e7ea4-4566-40e0-b29e-7fe767bb895b","Type":"ContainerDied","Data":"7fbd239c4ea3730acf00ef149a932fea60868b7bbb592965c482158dc8468dba"}
Sep 16 01:09:53.907785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.907716    2581 scope.go:117] "RemoveContainer" containerID="c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0"
Sep 16 01:09:53.917518 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.917494    2581 scope.go:117] "RemoveContainer" containerID="c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0"
Sep 16 01:09:53.917816 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:09:53.917796    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0\": container with ID starting with c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0 not found: ID does not exist" containerID="c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0"
Sep 16 01:09:53.917888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.917823    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0"} err="failed to get container status \"c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0\": rpc error: code = NotFound desc = could not find container \"c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0\": container with ID starting with c9f8f7379df63df19de985ffd11a04c1aeb72f7ea092e0a9a017e932f40fcfa0 not found: ID does not exist"
Sep 16 01:09:53.928399 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.928370    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"]
Sep 16 01:09:53.932099 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.932069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:53.934806 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.934785    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f29ccae3-6786c86459-z7wbd"]
Sep 16 01:09:53.941161 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:53.941110    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36e7ea4-4566-40e0-b29e-7fe767bb895b" path="/var/lib/kubelet/pods/f36e7ea4-4566-40e0-b29e-7fe767bb895b/volumes"
Sep 16 01:09:54.000345 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:54.000257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:54.101662 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:54.101622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:54.316415 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:54.316383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:54.811607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:54.811574    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:55.717590 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:55.717550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:57.541079 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.541028    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x" podUID="5b50e2f6-477d-4780-8242-b8a13f011411" containerName="main" containerID="cri-o://a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e" gracePeriod=45
Sep 16 01:09:57.790548 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.790519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:09:57.812209 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-model-cache\") pod \"5b50e2f6-477d-4780-8242-b8a13f011411\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") "
Sep 16 01:09:57.812209 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-tmp-dir\") pod \"5b50e2f6-477d-4780-8242-b8a13f011411\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") "
Sep 16 01:09:57.812209 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812138    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-home\") pod \"5b50e2f6-477d-4780-8242-b8a13f011411\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") "
Sep 16 01:09:57.812209 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b50e2f6-477d-4780-8242-b8a13f011411-tls-certs\") pod \"5b50e2f6-477d-4780-8242-b8a13f011411\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") "
Sep 16 01:09:57.812614 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812235    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdzsq\" (UniqueName: \"kubernetes.io/projected/5b50e2f6-477d-4780-8242-b8a13f011411-kube-api-access-gdzsq\") pod \"5b50e2f6-477d-4780-8242-b8a13f011411\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") "
Sep 16 01:09:57.812614 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-dshm\") pod \"5b50e2f6-477d-4780-8242-b8a13f011411\" (UID: \"5b50e2f6-477d-4780-8242-b8a13f011411\") "
Sep 16 01:09:57.812614 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b50e2f6-477d-4780-8242-b8a13f011411" (UID: "5b50e2f6-477d-4780-8242-b8a13f011411"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.812614 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812388    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b50e2f6-477d-4780-8242-b8a13f011411" (UID: "5b50e2f6-477d-4780-8242-b8a13f011411"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.812843 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-home" (OuterVolumeSpecName: "home") pod "5b50e2f6-477d-4780-8242-b8a13f011411" (UID: "5b50e2f6-477d-4780-8242-b8a13f011411"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.812843 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812701    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-model-cache\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.812843 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.812722    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-tmp-dir\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.815302 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.815226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b50e2f6-477d-4780-8242-b8a13f011411-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b50e2f6-477d-4780-8242-b8a13f011411" (UID: "5b50e2f6-477d-4780-8242-b8a13f011411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:09:57.815302 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.815252    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-dshm" (OuterVolumeSpecName: "dshm") pod "5b50e2f6-477d-4780-8242-b8a13f011411" (UID: "5b50e2f6-477d-4780-8242-b8a13f011411"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.815557 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.815460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b50e2f6-477d-4780-8242-b8a13f011411-kube-api-access-gdzsq" (OuterVolumeSpecName: "kube-api-access-gdzsq") pod "5b50e2f6-477d-4780-8242-b8a13f011411" (UID: "5b50e2f6-477d-4780-8242-b8a13f011411"). InnerVolumeSpecName "kube-api-access-gdzsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:09:57.914213 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.914178    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-dshm\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.914213 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.914208    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b50e2f6-477d-4780-8242-b8a13f011411-home\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.914425 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.914224    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b50e2f6-477d-4780-8242-b8a13f011411-tls-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.914425 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.914235    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdzsq\" (UniqueName: \"kubernetes.io/projected/5b50e2f6-477d-4780-8242-b8a13f011411-kube-api-access-gdzsq\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.930123 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.930088    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b50e2f6-477d-4780-8242-b8a13f011411" containerID="a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e" exitCode=0
Sep 16 01:09:57.930299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.930138    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"
Sep 16 01:09:57.930299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.930177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x" event={"ID":"5b50e2f6-477d-4780-8242-b8a13f011411","Type":"ContainerDied","Data":"a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e"}
Sep 16 01:09:57.930299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.930224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x" event={"ID":"5b50e2f6-477d-4780-8242-b8a13f011411","Type":"ContainerDied","Data":"f34867e95501e697a4e9c63f6eac9dc82c053d99d57b40bde1d817a38442b62b"}
Sep 16 01:09:57.930299 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.930246    2581 scope.go:117] "RemoveContainer" containerID="a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e"
Sep 16 01:09:57.941241 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.941221    2581 scope.go:117] "RemoveContainer" containerID="a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e"
Sep 16 01:09:57.941529 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:09:57.941506    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e\": container with ID starting with a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e not found: ID does not exist" containerID="a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e"
Sep 16 01:09:57.941621 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.941542    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e"} err="failed to get container status \"a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e\": rpc error: code = NotFound desc = could not find container \"a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e\": container with ID starting with a9e35eae6a24259e5cb13a3f9d59d67652106120e954e7a434ba92012af57e9e not found: ID does not exist"
Sep 16 01:09:57.951315 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.951292    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"]
Sep 16 01:09:57.954794 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:57.954773    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/simulator-w0-96c840-kserve-7d445594f7-82z9x"]
Sep 16 01:09:59.758352 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:59.758320    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:59.861235 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:59.861200    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:09:59.942333 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:59.942288    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b50e2f6-477d-4780-8242-b8a13f011411" path="/var/lib/kubelet/pods/5b50e2f6-477d-4780-8242-b8a13f011411/volumes"
Sep 16 01:09:59.999663 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:59.999628    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"]
Sep 16 01:09:59.999880 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:09:59.999853    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" containerID="cri-o://9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755" gracePeriod=30
Sep 16 01:10:00.021913 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:00.021884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:00.320434 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:00.320347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:00.423395 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:00.423359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:00.524839 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:00.524804    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:00.592156 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:00.592062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:00.653162 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:00.653108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:03.233404 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:03.233354    2581 patch_prober.go:28] interesting pod/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 01:10:03.233815 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:03.233419    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 01:10:05.251108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.251081    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:10:05.388846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-data\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.388846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388792    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-socket\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.388846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkcm5\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-kube-api-access-bkcm5\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-certs\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-credential-socket\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-token\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-podinfo\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.388982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istiod-ca-cert\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-envoy\") pod \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\" (UID: \"d36c2b28-4782-4560-bfb2-5e82fe2c2c44\") "
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-data" (OuterVolumeSpecName: "istio-data") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.389143 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389131    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.389536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.389536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.389536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389393    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.389536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389415    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.389536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389430    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.389536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389444    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.389842 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.389741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:10:05.391281 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.391254    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-token" (OuterVolumeSpecName: "istio-token") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:10:05.391401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.391286    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-kube-api-access-bkcm5" (OuterVolumeSpecName: "kube-api-access-bkcm5") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "kube-api-access-bkcm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:10:05.391401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.391362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.391401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.391374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d36c2b28-4782-4560-bfb2-5e82fe2c2c44" (UID: "d36c2b28-4782-4560-bfb2-5e82fe2c2c44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:10:05.490191 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.490150    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkcm5\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-kube-api-access-bkcm5\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.490191 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.490184    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.490191 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.490194    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.490191 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.490201    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.490474 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.490209    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36c2b28-4782-4560-bfb2-5e82fe2c2c44-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.974915 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.974883    2581 generic.go:356] "Generic (PLEG): container finished" podID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerID="9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755" exitCode=0
Sep 16 01:10:05.975082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.974937    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"
Sep 16 01:10:05.975082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.974970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" event={"ID":"d36c2b28-4782-4560-bfb2-5e82fe2c2c44","Type":"ContainerDied","Data":"9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755"}
Sep 16 01:10:05.975082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.975030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t" event={"ID":"d36c2b28-4782-4560-bfb2-5e82fe2c2c44","Type":"ContainerDied","Data":"d3ebfec69bb046a7138ad1b50256abd495a7df2c05744694ffc329b4f785654c"}
Sep 16 01:10:05.975082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.975053    2581 scope.go:117] "RemoveContainer" containerID="9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755"
Sep 16 01:10:05.984607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.984586    2581 scope.go:117] "RemoveContainer" containerID="6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2"
Sep 16 01:10:05.992169 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.992139    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"]
Sep 16 01:10:05.993957 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.993932    2581 scope.go:117] "RemoveContainer" containerID="9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755"
Sep 16 01:10:05.994314 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:10:05.994287    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755\": container with ID starting with 9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755 not found: ID does not exist" containerID="9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755"
Sep 16 01:10:05.994370 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.994323    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755"} err="failed to get container status \"9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755\": rpc error: code = NotFound desc = could not find container \"9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755\": container with ID starting with 9a40de56702ff0a77ce21f7e9549adc7bcb87f5cdd0a803ca4ff8bd360067755 not found: ID does not exist"
Sep 16 01:10:05.994370 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.994341    2581 scope.go:117] "RemoveContainer" containerID="6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2"
Sep 16 01:10:05.994752 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:10:05.994726    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2\": container with ID starting with 6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2 not found: ID does not exist" containerID="6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2"
Sep 16 01:10:05.994827 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.994762    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2"} err="failed to get container status \"6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2\": rpc error: code = NotFound desc = could not find container \"6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2\": container with ID starting with 6a81e015100d6e7bc9e4b74e034887062f3554d5d7231d98c42a487a3ac22ca2 not found: ID does not exist"
Sep 16 01:10:05.995529 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:05.995506    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f29ccae3-openshift-default-5bb66cfc6-prw6t"]
Sep 16 01:10:06.281892 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:06.281854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:06.342026 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:06.341990    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:06.418224 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:06.418173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:06.510997 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:06.510961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:06.710538 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:06.710431    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:07.276493 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:07.276454    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:10:07.941186 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:07.941152    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" path="/var/lib/kubelet/pods/d36c2b28-4782-4560-bfb2-5e82fe2c2c44/volumes"
Sep 16 01:10:12.269873 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:12.269821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:12.332435 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:12.332393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:13.218846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:13.218806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:13.554199 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:13.554166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:13.902239 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:13.902139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:14.002246 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:14.002211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:16.699829 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:16.699790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:16.822696 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:16.822614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:16.914240 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:16.914201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:18.133739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:18.133702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:18.199456 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:18.199421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:18.565972 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:18.565939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:18.667241 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:18.667184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:18.742637 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:18.742602    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:18.813406 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:18.813373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:20.103441 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:20.103404    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:22.794907 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:22.794864    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:10:25.669710 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:25.669660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:31.119738 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:31.119695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:39.998918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:39.998874    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:40.900694 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:40.900656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:41.577100 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:41.577066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:41.817618 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:41.817585    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:42.160445 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.160412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:42.444869 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.444782    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"]
Sep 16 01:10:42.445286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445249    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36e7ea4-4566-40e0-b29e-7fe767bb895b" containerName="maas-api"
Sep 16 01:10:42.445286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36e7ea4-4566-40e0-b29e-7fe767bb895b" containerName="maas-api"
Sep 16 01:10:42.445286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445287    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy"
Sep 16 01:10:42.445506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445296    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy"
Sep 16 01:10:42.445506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445324    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b50e2f6-477d-4780-8242-b8a13f011411" containerName="main"
Sep 16 01:10:42.445506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445332    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b50e2f6-477d-4780-8242-b8a13f011411" containerName="main"
Sep 16 01:10:42.445506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445440    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy"
Sep 16 01:10:42.445506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445451    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b50e2f6-477d-4780-8242-b8a13f011411" containerName="main"
Sep 16 01:10:42.445506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445461    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36e7ea4-4566-40e0-b29e-7fe767bb895b" containerName="maas-api"
Sep 16 01:10:42.445506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.445473    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy"
Sep 16 01:10:42.448818 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.448799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.451117 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.451094    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 01:10:42.451249 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.451225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-45446b-openshift-default-dockercfg-w9558\""
Sep 16 01:10:42.461692 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.461661    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"]
Sep 16 01:10:42.473531 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.473493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:42.528048 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-envoy\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528241 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-credential-socket\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528241 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f419f-4ff6-4511-801f-e50af17337b3-istiod-ca-cert\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528241 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-certs\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528241 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-socket\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528241 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mt8\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-kube-api-access-d6mt8\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528514 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-data\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528514 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-istio-token\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.528514 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.528341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f419f-4ff6-4511-801f-e50af17337b3-istio-podinfo\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629496 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-envoy\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-credential-socket\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f419f-4ff6-4511-801f-e50af17337b3-istiod-ca-cert\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-certs\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-socket\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mt8\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-kube-api-access-d6mt8\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-data\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-istio-token\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f419f-4ff6-4511-801f-e50af17337b3-istio-podinfo\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.629990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.629993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-certs\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.630529 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.630045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-socket\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.630529 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.630301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-credential-socket\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.630529 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.630380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f419f-4ff6-4511-801f-e50af17337b3-istiod-ca-cert\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.630529 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.630412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-data\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.637541 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.632184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-envoy\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.637541 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.632661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f419f-4ff6-4511-801f-e50af17337b3-istio-podinfo\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.643221 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.643185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mt8\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-kube-api-access-d6mt8\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.643383 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.643199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-istio-token\") pod \"e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") " pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.762415 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.762388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:42.793611 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.793577    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:42.889500 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:42.889456    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:10:43.038670 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.038639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"]
Sep 16 01:10:43.040262 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:10:43.040229    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484f419f_4ff6_4511_801f_e50af17337b3.slice/crio-6f95efa9bf9374a6640a48836bf10a3218dd6307cd38d95f7cf524c5a7ded770 WatchSource:0}: Error finding container 6f95efa9bf9374a6640a48836bf10a3218dd6307cd38d95f7cf524c5a7ded770: Status 404 returned error can't find the container with id 6f95efa9bf9374a6640a48836bf10a3218dd6307cd38d95f7cf524c5a7ded770
Sep 16 01:10:43.042563 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.042525    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:10:43.042656 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.042610    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:10:43.042732 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.042651    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:10:43.164716 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.164675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" event={"ID":"484f419f-4ff6-4511-801f-e50af17337b3","Type":"ContainerStarted","Data":"532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e"}
Sep 16 01:10:43.164884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.164741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" event={"ID":"484f419f-4ff6-4511-801f-e50af17337b3","Type":"ContainerStarted","Data":"6f95efa9bf9374a6640a48836bf10a3218dd6307cd38d95f7cf524c5a7ded770"}
Sep 16 01:10:43.183988 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.183633    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podStartSLOduration=1.183608367 podStartE2EDuration="1.183608367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:10:43.182187639 +0000 UTC m=+1661.725324614" watchObservedRunningTime="2026-09-16 01:10:43.183608367 +0000 UTC m=+1661.726745343"
Sep 16 01:10:43.762559 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:43.762522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:44.768461 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:44.768429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:45.079914 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:45.079816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:45.175458 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:45.175427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:45.176517 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:45.176498    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:10:45.188465 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:45.188433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:45.288928 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:45.288896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:46.086209 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:46.086176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:47.026416 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:47.026381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:47.126623 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:47.126590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:47.581630 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:47.581594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:47.686134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:47.686101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:47.746720 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:47.746689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:48.056375 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:48.056344    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:48.928782 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:48.928745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:49.651962 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:49.651928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:10:51.018044 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:10:51.018007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:00.778827 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:00.778789    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:11:01.217716 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:01.217624    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 01:11:01.217716 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:01.217704    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 01:11:02.267066 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:02.267033    2581 generic.go:356] "Generic (PLEG): container finished" podID="484f419f-4ff6-4511-801f-e50af17337b3" containerID="532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e" exitCode=0
Sep 16 01:11:02.267533 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:02.267086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" event={"ID":"484f419f-4ff6-4511-801f-e50af17337b3","Type":"ContainerDied","Data":"532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e"}
Sep 16 01:11:02.267533 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:02.267469    2581 scope.go:117] "RemoveContainer" containerID="532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e"
Sep 16 01:11:02.268643 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:02.268598    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:11:02.268776 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:02.268691    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:11:02.268776 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:02.268740    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:11:03.278446 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:03.278403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" event={"ID":"484f419f-4ff6-4511-801f-e50af17337b3","Type":"ContainerStarted","Data":"5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a"}
Sep 16 01:11:03.762847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:03.762809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:11:03.763056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:03.762941    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 01:11:03.763056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:03.762986    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 01:11:04.763286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:04.763244    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 01:11:04.763934 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:04.763900    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 01:11:05.763466 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:05.763424    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 01:11:05.764081 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:05.764046    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 01:11:06.763149 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:06.763099    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 01:11:06.763367 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:06.763171    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 01:11:07.763393 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:07.763348    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 01:11:07.763933 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:07.763425    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 01:11:08.767881 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:08.767846    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:11:08.768373 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:08.768241    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:11:08.768982 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:08.768961    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:11:09.229351 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:09.229257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:10.121004 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.120971    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:10.221835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.221800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:10.838540 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.838500    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:10.978491 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.978444    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9040-259589"]
Sep 16 01:11:10.978859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.978846    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy"
Sep 16 01:11:10.978904 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.978860    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36c2b28-4782-4560-bfb2-5e82fe2c2c44" containerName="istio-proxy"
Sep 16 01:11:10.980884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.980868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:10.983393 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.983365    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c6q92\""
Sep 16 01:11:10.986259 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:10.986239    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-259589"]
Sep 16 01:11:11.120374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:11.120286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbdn\" (UniqueName: \"kubernetes.io/projected/41ccb6ad-d125-4f5f-855e-7c1151fdc243-kube-api-access-tgbdn\") pod \"test-curl-9040-259589\" (UID: \"41ccb6ad-d125-4f5f-855e-7c1151fdc243\") " pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:11.221256 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:11.221217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbdn\" (UniqueName: \"kubernetes.io/projected/41ccb6ad-d125-4f5f-855e-7c1151fdc243-kube-api-access-tgbdn\") pod \"test-curl-9040-259589\" (UID: \"41ccb6ad-d125-4f5f-855e-7c1151fdc243\") " pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:11.229036 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:11.229002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbdn\" (UniqueName: \"kubernetes.io/projected/41ccb6ad-d125-4f5f-855e-7c1151fdc243-kube-api-access-tgbdn\") pod \"test-curl-9040-259589\" (UID: \"41ccb6ad-d125-4f5f-855e-7c1151fdc243\") " pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:11.292437 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:11.292384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:11.559840 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:11.559677    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-259589"]
Sep 16 01:11:11.562136 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:11:11.562106    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41ccb6ad_d125_4f5f_855e_7c1151fdc243.slice/crio-b2403d60d927e63fd292a7317fcdb2ca17e7f9c5672b1c3ab71d0b572ce794b7 WatchSource:0}: Error finding container b2403d60d927e63fd292a7317fcdb2ca17e7f9c5672b1c3ab71d0b572ce794b7: Status 404 returned error can't find the container with id b2403d60d927e63fd292a7317fcdb2ca17e7f9c5672b1c3ab71d0b572ce794b7
Sep 16 01:11:11.642539 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:11.642513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:12.334438 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:12.334398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-259589" event={"ID":"41ccb6ad-d125-4f5f-855e-7c1151fdc243","Type":"ContainerStarted","Data":"727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8"}
Sep 16 01:11:12.334438 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:12.334441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-259589" event={"ID":"41ccb6ad-d125-4f5f-855e-7c1151fdc243","Type":"ContainerStarted","Data":"b2403d60d927e63fd292a7317fcdb2ca17e7f9c5672b1c3ab71d0b572ce794b7"}
Sep 16 01:11:12.349911 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:12.349858    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9040-259589" podStartSLOduration=2.349843323 podStartE2EDuration="2.349843323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:12.346917697 +0000 UTC m=+1690.890054672" watchObservedRunningTime="2026-09-16 01:11:12.349843323 +0000 UTC m=+1690.892980298"
Sep 16 01:11:12.935281 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:12.935234    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9040-259589"]
Sep 16 01:11:12.940779 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:12.940745    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9040-259589"]
Sep 16 01:11:13.062136 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.062099    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9040-7bfdb0"]
Sep 16 01:11:13.062524 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.062507    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ccb6ad-d125-4f5f-855e-7c1151fdc243" containerName="test-curl-9040-259589"
Sep 16 01:11:13.062576 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.062526    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ccb6ad-d125-4f5f-855e-7c1151fdc243" containerName="test-curl-9040-259589"
Sep 16 01:11:13.062613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.062597    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ccb6ad-d125-4f5f-855e-7c1151fdc243" containerName="test-curl-9040-259589"
Sep 16 01:11:13.064657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.064641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:13.071012 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.070981    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-7bfdb0"]
Sep 16 01:11:13.134384 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.134345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bf5f\" (UniqueName: \"kubernetes.io/projected/5ad6a57a-6d44-4d08-80a3-7c2543c924bb-kube-api-access-5bf5f\") pod \"test-curl-9040-7bfdb0\" (UID: \"5ad6a57a-6d44-4d08-80a3-7c2543c924bb\") " pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:13.234974 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.234938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf5f\" (UniqueName: \"kubernetes.io/projected/5ad6a57a-6d44-4d08-80a3-7c2543c924bb-kube-api-access-5bf5f\") pod \"test-curl-9040-7bfdb0\" (UID: \"5ad6a57a-6d44-4d08-80a3-7c2543c924bb\") " pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:13.242776 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.242750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bf5f\" (UniqueName: \"kubernetes.io/projected/5ad6a57a-6d44-4d08-80a3-7c2543c924bb-kube-api-access-5bf5f\") pod \"test-curl-9040-7bfdb0\" (UID: \"5ad6a57a-6d44-4d08-80a3-7c2543c924bb\") " pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:13.339544 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:13.339435    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9040-259589" container="test-curl-9040-259589"
Sep 16 01:11:13.376164 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.376113    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:13.644345 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:13.644316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-7bfdb0"]
Sep 16 01:11:13.646059 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:11:13.646034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ad6a57a_6d44_4d08_80a3_7c2543c924bb.slice/crio-7dc69e73b9e482fad6b73c42bf22b6f6c2dc51180c6d88068aead009d66a67c7 WatchSource:0}: Error finding container 7dc69e73b9e482fad6b73c42bf22b6f6c2dc51180c6d88068aead009d66a67c7: Status 404 returned error can't find the container with id 7dc69e73b9e482fad6b73c42bf22b6f6c2dc51180c6d88068aead009d66a67c7
Sep 16 01:11:14.346670 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:14.346623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-7bfdb0" event={"ID":"5ad6a57a-6d44-4d08-80a3-7c2543c924bb","Type":"ContainerStarted","Data":"40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc"}
Sep 16 01:11:14.347144 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:14.346677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-7bfdb0" event={"ID":"5ad6a57a-6d44-4d08-80a3-7c2543c924bb","Type":"ContainerStarted","Data":"7dc69e73b9e482fad6b73c42bf22b6f6c2dc51180c6d88068aead009d66a67c7"}
Sep 16 01:11:14.347144 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:14.346783    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9040-259589" podUID="41ccb6ad-d125-4f5f-855e-7c1151fdc243" containerName="test-curl-9040-259589" containerID="cri-o://727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8" gracePeriod=2
Sep 16 01:11:14.360351 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:14.360301    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9040-7bfdb0" podStartSLOduration=1.3602833269999999 podStartE2EDuration="1.360283327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:14.359252843 +0000 UTC m=+1692.902389818" watchObservedRunningTime="2026-09-16 01:11:14.360283327 +0000 UTC m=+1692.903420299"
Sep 16 01:11:14.957655 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:14.957615    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9040-7bfdb0"]
Sep 16 01:11:14.962011 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:14.961979    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9040-7bfdb0"]
Sep 16 01:11:15.083940 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.083904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9040-1c8c55"]
Sep 16 01:11:15.084337 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.084321    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" containerName="test-curl-9040-7bfdb0"
Sep 16 01:11:15.084397 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.084340    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" containerName="test-curl-9040-7bfdb0"
Sep 16 01:11:15.084434 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.084416    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" containerName="test-curl-9040-7bfdb0"
Sep 16 01:11:15.086536 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.086520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:15.090795 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.090768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-1c8c55"]
Sep 16 01:11:15.152397 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.152356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxl7\" (UniqueName: \"kubernetes.io/projected/94e50836-54ff-4d5e-be51-b3ed179092af-kube-api-access-zgxl7\") pod \"test-curl-9040-1c8c55\" (UID: \"94e50836-54ff-4d5e-be51-b3ed179092af\") " pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:15.252953 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.252918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxl7\" (UniqueName: \"kubernetes.io/projected/94e50836-54ff-4d5e-be51-b3ed179092af-kube-api-access-zgxl7\") pod \"test-curl-9040-1c8c55\" (UID: \"94e50836-54ff-4d5e-be51-b3ed179092af\") " pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:15.261336 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.261300    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxl7\" (UniqueName: \"kubernetes.io/projected/94e50836-54ff-4d5e-be51-b3ed179092af-kube-api-access-zgxl7\") pod \"test-curl-9040-1c8c55\" (UID: \"94e50836-54ff-4d5e-be51-b3ed179092af\") " pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:15.352495 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:15.352403    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9040-7bfdb0" container="test-curl-9040-7bfdb0"
Sep 16 01:11:15.396561 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.396518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:15.663519 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:15.663475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-1c8c55"]
Sep 16 01:11:15.665502 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:11:15.665459    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e50836_54ff_4d5e_be51_b3ed179092af.slice/crio-176dcdf690e5bb20b57172961facbb1be7d579f52efc63928bb731b0f9ee721e WatchSource:0}: Error finding container 176dcdf690e5bb20b57172961facbb1be7d579f52efc63928bb731b0f9ee721e: Status 404 returned error can't find the container with id 176dcdf690e5bb20b57172961facbb1be7d579f52efc63928bb731b0f9ee721e
Sep 16 01:11:16.359514 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.359464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-1c8c55" event={"ID":"94e50836-54ff-4d5e-be51-b3ed179092af","Type":"ContainerStarted","Data":"3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67"}
Sep 16 01:11:16.359851 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.359530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-1c8c55" event={"ID":"94e50836-54ff-4d5e-be51-b3ed179092af","Type":"ContainerStarted","Data":"176dcdf690e5bb20b57172961facbb1be7d579f52efc63928bb731b0f9ee721e"}
Sep 16 01:11:16.359851 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.359694    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9040-7bfdb0" podUID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" containerName="test-curl-9040-7bfdb0" containerID="cri-o://40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc" gracePeriod=2
Sep 16 01:11:16.373395 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.373347    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9040-1c8c55" podStartSLOduration=1.373330192 podStartE2EDuration="1.373330192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:16.371573709 +0000 UTC m=+1694.914710685" watchObservedRunningTime="2026-09-16 01:11:16.373330192 +0000 UTC m=+1694.916467168"
Sep 16 01:11:16.509809 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.509783    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:16.563405 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.563370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgbdn\" (UniqueName: \"kubernetes.io/projected/41ccb6ad-d125-4f5f-855e-7c1151fdc243-kube-api-access-tgbdn\") pod \"41ccb6ad-d125-4f5f-855e-7c1151fdc243\" (UID: \"41ccb6ad-d125-4f5f-855e-7c1151fdc243\") "
Sep 16 01:11:16.565528 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.565500    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ccb6ad-d125-4f5f-855e-7c1151fdc243-kube-api-access-tgbdn" (OuterVolumeSpecName: "kube-api-access-tgbdn") pod "41ccb6ad-d125-4f5f-855e-7c1151fdc243" (UID: "41ccb6ad-d125-4f5f-855e-7c1151fdc243"). InnerVolumeSpecName "kube-api-access-tgbdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:11:16.664923 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.664832    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgbdn\" (UniqueName: \"kubernetes.io/projected/41ccb6ad-d125-4f5f-855e-7c1151fdc243-kube-api-access-tgbdn\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:16.882119 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.882083    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:11:16.966748 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.966659    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9040-1c8c55"]
Sep 16 01:11:16.971621 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:16.971592    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9040-1c8c55"]
Sep 16 01:11:17.100997 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.100960    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9040-4992eb"]
Sep 16 01:11:17.101389 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.101377    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e50836-54ff-4d5e-be51-b3ed179092af" containerName="test-curl-9040-1c8c55"
Sep 16 01:11:17.101429 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.101391    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e50836-54ff-4d5e-be51-b3ed179092af" containerName="test-curl-9040-1c8c55"
Sep 16 01:11:17.101500 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.101491    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e50836-54ff-4d5e-be51-b3ed179092af" containerName="test-curl-9040-1c8c55"
Sep 16 01:11:17.103606 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.103589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:17.109883 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.109851    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-4992eb"]
Sep 16 01:11:17.169609 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.169572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8zb\" (UniqueName: \"kubernetes.io/projected/b452b594-9f74-4e6b-9e80-69c32e53ca43-kube-api-access-hc8zb\") pod \"test-curl-9040-4992eb\" (UID: \"b452b594-9f74-4e6b-9e80-69c32e53ca43\") " pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:17.270312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.270261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8zb\" (UniqueName: \"kubernetes.io/projected/b452b594-9f74-4e6b-9e80-69c32e53ca43-kube-api-access-hc8zb\") pod \"test-curl-9040-4992eb\" (UID: \"b452b594-9f74-4e6b-9e80-69c32e53ca43\") " pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:17.280699 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.280602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8zb\" (UniqueName: \"kubernetes.io/projected/b452b594-9f74-4e6b-9e80-69c32e53ca43-kube-api-access-hc8zb\") pod \"test-curl-9040-4992eb\" (UID: \"b452b594-9f74-4e6b-9e80-69c32e53ca43\") " pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:17.366393 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.366350    2581 generic.go:356] "Generic (PLEG): container finished" podID="41ccb6ad-d125-4f5f-855e-7c1151fdc243" containerID="727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8" exitCode=137
Sep 16 01:11:17.366820 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.366455    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:17.366820 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.366517    2581 scope.go:117] "RemoveContainer" containerID="727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8"
Sep 16 01:11:17.366820 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:17.366775    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9040-1c8c55" container="test-curl-9040-1c8c55"
Sep 16 01:11:17.376735 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.376703    2581 scope.go:117] "RemoveContainer" containerID="727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8"
Sep 16 01:11:17.377107 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:17.377073    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8\": container with ID starting with 727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8 not found: ID does not exist" containerID="727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8"
Sep 16 01:11:17.377202 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.377127    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8"} err="failed to get container status \"727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8\": rpc error: code = NotFound desc = could not find container \"727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8\": container with ID starting with 727334f491c95f5f1ace2ce7f649f16d48c6b253427beef702205a64527301e8 not found: ID does not exist"
Sep 16 01:11:17.378716 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:17.378689    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9040-259589\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="41ccb6ad-d125-4f5f-855e-7c1151fdc243" pod="opendatahub/test-curl-9040-259589"
Sep 16 01:11:17.413991 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.413956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:17.688887 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.688857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9040-4992eb"]
Sep 16 01:11:17.691144 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:11:17.691114    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb452b594_9f74_4e6b_9e80_69c32e53ca43.slice/crio-f8b1fae84bfe30df3bf1a3dafd2ae89e3a68809e12c391e32e2a46f0057fbfc7 WatchSource:0}: Error finding container f8b1fae84bfe30df3bf1a3dafd2ae89e3a68809e12c391e32e2a46f0057fbfc7: Status 404 returned error can't find the container with id f8b1fae84bfe30df3bf1a3dafd2ae89e3a68809e12c391e32e2a46f0057fbfc7
Sep 16 01:11:17.942255 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:17.942174    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ccb6ad-d125-4f5f-855e-7c1151fdc243" path="/var/lib/kubelet/pods/41ccb6ad-d125-4f5f-855e-7c1151fdc243/volumes"
Sep 16 01:11:18.373858 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.373826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-4992eb" event={"ID":"b452b594-9f74-4e6b-9e80-69c32e53ca43","Type":"ContainerStarted","Data":"117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295"}
Sep 16 01:11:18.374240 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.373884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9040-4992eb" event={"ID":"b452b594-9f74-4e6b-9e80-69c32e53ca43","Type":"ContainerStarted","Data":"f8b1fae84bfe30df3bf1a3dafd2ae89e3a68809e12c391e32e2a46f0057fbfc7"}
Sep 16 01:11:18.374970 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.374944    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9040-1c8c55" podUID="94e50836-54ff-4d5e-be51-b3ed179092af" containerName="test-curl-9040-1c8c55" containerID="cri-o://3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67" gracePeriod=2
Sep 16 01:11:18.388575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.388526    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9040-4992eb" podStartSLOduration=1.388510807 podStartE2EDuration="1.388510807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:18.386444416 +0000 UTC m=+1696.929581391" watchObservedRunningTime="2026-09-16 01:11:18.388510807 +0000 UTC m=+1696.931647776"
Sep 16 01:11:18.526419 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.526389    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:18.582946 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.582918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bf5f\" (UniqueName: \"kubernetes.io/projected/5ad6a57a-6d44-4d08-80a3-7c2543c924bb-kube-api-access-5bf5f\") pod \"5ad6a57a-6d44-4d08-80a3-7c2543c924bb\" (UID: \"5ad6a57a-6d44-4d08-80a3-7c2543c924bb\") "
Sep 16 01:11:18.585110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.585073    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad6a57a-6d44-4d08-80a3-7c2543c924bb-kube-api-access-5bf5f" (OuterVolumeSpecName: "kube-api-access-5bf5f") pod "5ad6a57a-6d44-4d08-80a3-7c2543c924bb" (UID: "5ad6a57a-6d44-4d08-80a3-7c2543c924bb"). InnerVolumeSpecName "kube-api-access-5bf5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:11:18.683964 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.683874    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bf5f\" (UniqueName: \"kubernetes.io/projected/5ad6a57a-6d44-4d08-80a3-7c2543c924bb-kube-api-access-5bf5f\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:18.970824 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.970715    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9040-4992eb"]
Sep 16 01:11:18.973106 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:18.973077    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9040-4992eb"]
Sep 16 01:11:19.382125 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.382085    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" containerID="40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc" exitCode=137
Sep 16 01:11:19.382627 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.382141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:19.382627 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.382182    2581 scope.go:117] "RemoveContainer" containerID="40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc"
Sep 16 01:11:19.382627 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:19.382582    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9040-4992eb" container="test-curl-9040-4992eb"
Sep 16 01:11:19.391801 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.391750    2581 scope.go:117] "RemoveContainer" containerID="40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc"
Sep 16 01:11:19.392034 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:19.392013    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc\": container with ID starting with 40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc not found: ID does not exist" containerID="40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc"
Sep 16 01:11:19.392118 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.392041    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc"} err="failed to get container status \"40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc\": rpc error: code = NotFound desc = could not find container \"40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc\": container with ID starting with 40782b3c1f6115e5e7612784af7be43266fddce4c7cf0007212b8c07e3014ebc not found: ID does not exist"
Sep 16 01:11:19.392952 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:19.392926    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9040-7bfdb0\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:19.596752 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.596709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:19.747965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.747882    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9043-e1b522"]
Sep 16 01:11:19.748357 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.748343    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b452b594-9f74-4e6b-9e80-69c32e53ca43" containerName="test-curl-9040-4992eb"
Sep 16 01:11:19.748397 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.748360    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b452b594-9f74-4e6b-9e80-69c32e53ca43" containerName="test-curl-9040-4992eb"
Sep 16 01:11:19.748433 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.748419    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b452b594-9f74-4e6b-9e80-69c32e53ca43" containerName="test-curl-9040-4992eb"
Sep 16 01:11:19.751415 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.751388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:19.753463 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:19.753419    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9040-7bfdb0\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" pod="opendatahub/test-curl-9040-7bfdb0"
Sep 16 01:11:19.754464 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.754440    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9043-e1b522"]
Sep 16 01:11:19.795976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.795947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjqk\" (UniqueName: \"kubernetes.io/projected/ca0d8d5f-6283-4ac6-95c4-41a57db0cc79-kube-api-access-jnjqk\") pod \"test-curl-9043-e1b522\" (UID: \"ca0d8d5f-6283-4ac6-95c4-41a57db0cc79\") " pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:19.896926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.896888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjqk\" (UniqueName: \"kubernetes.io/projected/ca0d8d5f-6283-4ac6-95c4-41a57db0cc79-kube-api-access-jnjqk\") pod \"test-curl-9043-e1b522\" (UID: \"ca0d8d5f-6283-4ac6-95c4-41a57db0cc79\") " pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:19.906399 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.906362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:19.907654 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.907631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjqk\" (UniqueName: \"kubernetes.io/projected/ca0d8d5f-6283-4ac6-95c4-41a57db0cc79-kube-api-access-jnjqk\") pod \"test-curl-9043-e1b522\" (UID: \"ca0d8d5f-6283-4ac6-95c4-41a57db0cc79\") " pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:19.941682 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:19.941653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ad6a57a-6d44-4d08-80a3-7c2543c924bb" path="/var/lib/kubelet/pods/5ad6a57a-6d44-4d08-80a3-7c2543c924bb/volumes"
Sep 16 01:11:20.005475 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.005381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:20.063374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.063339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:20.330951 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.330916    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9043-e1b522"]
Sep 16 01:11:20.333257 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:11:20.333231    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca0d8d5f_6283_4ac6_95c4_41a57db0cc79.slice/crio-7f92675a1b6c3003a0dd9a70b3a26bf3370853072799d060e3104e34d888f560 WatchSource:0}: Error finding container 7f92675a1b6c3003a0dd9a70b3a26bf3370853072799d060e3104e34d888f560: Status 404 returned error can't find the container with id 7f92675a1b6c3003a0dd9a70b3a26bf3370853072799d060e3104e34d888f560
Sep 16 01:11:20.390552 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.390517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9043-e1b522" event={"ID":"ca0d8d5f-6283-4ac6-95c4-41a57db0cc79","Type":"ContainerStarted","Data":"7f92675a1b6c3003a0dd9a70b3a26bf3370853072799d060e3104e34d888f560"}
Sep 16 01:11:20.390906 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.390647    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9040-4992eb" podUID="b452b594-9f74-4e6b-9e80-69c32e53ca43" containerName="test-curl-9040-4992eb" containerID="cri-o://117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295" gracePeriod=2
Sep 16 01:11:20.543820 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.543791    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:20.603298 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.603210    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgxl7\" (UniqueName: \"kubernetes.io/projected/94e50836-54ff-4d5e-be51-b3ed179092af-kube-api-access-zgxl7\") pod \"94e50836-54ff-4d5e-be51-b3ed179092af\" (UID: \"94e50836-54ff-4d5e-be51-b3ed179092af\") "
Sep 16 01:11:20.605267 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.605236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e50836-54ff-4d5e-be51-b3ed179092af-kube-api-access-zgxl7" (OuterVolumeSpecName: "kube-api-access-zgxl7") pod "94e50836-54ff-4d5e-be51-b3ed179092af" (UID: "94e50836-54ff-4d5e-be51-b3ed179092af"). InnerVolumeSpecName "kube-api-access-zgxl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:11:20.704785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:20.704748    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgxl7\" (UniqueName: \"kubernetes.io/projected/94e50836-54ff-4d5e-be51-b3ed179092af-kube-api-access-zgxl7\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:21.396833 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.396797    2581 generic.go:356] "Generic (PLEG): container finished" podID="94e50836-54ff-4d5e-be51-b3ed179092af" containerID="3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67" exitCode=137
Sep 16 01:11:21.396833 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.396837    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:21.397367 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.396878    2581 scope.go:117] "RemoveContainer" containerID="3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67"
Sep 16 01:11:21.398633 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.398607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9043-e1b522" event={"ID":"ca0d8d5f-6283-4ac6-95c4-41a57db0cc79","Type":"ContainerStarted","Data":"d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d"}
Sep 16 01:11:21.405895 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.405877    2581 scope.go:117] "RemoveContainer" containerID="3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67"
Sep 16 01:11:21.406201 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:21.406182    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67\": container with ID starting with 3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67 not found: ID does not exist" containerID="3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67"
Sep 16 01:11:21.406271 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.406210    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67"} err="failed to get container status \"3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67\": rpc error: code = NotFound desc = could not find container \"3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67\": container with ID starting with 3a53f0ed91ebad1bb3a5b5c53358427ab8ce78bfe5bb829c696a8573525d6b67 not found: ID does not exist"
Sep 16 01:11:21.414381 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:21.414350    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9040-1c8c55\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="94e50836-54ff-4d5e-be51-b3ed179092af" pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:21.939510 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:21.939444    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9040-1c8c55\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="94e50836-54ff-4d5e-be51-b3ed179092af" pod="opendatahub/test-curl-9040-1c8c55"
Sep 16 01:11:21.941595 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.941571    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e50836-54ff-4d5e-be51-b3ed179092af" path="/var/lib/kubelet/pods/94e50836-54ff-4d5e-be51-b3ed179092af/volumes"
Sep 16 01:11:21.987213 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.987159    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9043-e1b522" podStartSLOduration=2.987136755 podStartE2EDuration="2.987136755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:21.412382969 +0000 UTC m=+1699.955519944" watchObservedRunningTime="2026-09-16 01:11:21.987136755 +0000 UTC m=+1700.530273729"
Sep 16 01:11:21.988366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.988343    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9043-e1b522"]
Sep 16 01:11:21.992052 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:21.992019    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9043-e1b522"]
Sep 16 01:11:22.407115 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:22.407090    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9043-e1b522" container="test-curl-9043-e1b522"
Sep 16 01:11:22.566086 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:22.566065    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:22.625350 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:22.625310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc8zb\" (UniqueName: \"kubernetes.io/projected/b452b594-9f74-4e6b-9e80-69c32e53ca43-kube-api-access-hc8zb\") pod \"b452b594-9f74-4e6b-9e80-69c32e53ca43\" (UID: \"b452b594-9f74-4e6b-9e80-69c32e53ca43\") "
Sep 16 01:11:22.627527 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:22.627473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b452b594-9f74-4e6b-9e80-69c32e53ca43-kube-api-access-hc8zb" (OuterVolumeSpecName: "kube-api-access-hc8zb") pod "b452b594-9f74-4e6b-9e80-69c32e53ca43" (UID: "b452b594-9f74-4e6b-9e80-69c32e53ca43"). InnerVolumeSpecName "kube-api-access-hc8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:11:22.726938 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:22.726847    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8zb\" (UniqueName: \"kubernetes.io/projected/b452b594-9f74-4e6b-9e80-69c32e53ca43-kube-api-access-hc8zb\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:22.809692 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:22.809648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:23.239101 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.239062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:23.347653 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.347616    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:23.413076 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.413039    2581 generic.go:356] "Generic (PLEG): container finished" podID="b452b594-9f74-4e6b-9e80-69c32e53ca43" containerID="117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295" exitCode=137
Sep 16 01:11:23.413543 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.413081    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:23.413543 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.413105    2581 scope.go:117] "RemoveContainer" containerID="117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295"
Sep 16 01:11:23.413543 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.413376    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9043-e1b522" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" containerName="test-curl-9043-e1b522" containerID="cri-o://d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d" gracePeriod=2
Sep 16 01:11:23.415516 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:23.415464    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9043-e1b522\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:23.423016 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.423000    2581 scope.go:117] "RemoveContainer" containerID="117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295"
Sep 16 01:11:23.423291 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:23.423273    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295\": container with ID starting with 117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295 not found: ID does not exist" containerID="117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295"
Sep 16 01:11:23.423339 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.423298    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295"} err="failed to get container status \"117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295\": rpc error: code = NotFound desc = could not find container \"117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295\": container with ID starting with 117ccbc312760f901770627a7e88e14afa42770cad2f94ccf801203683c72295 not found: ID does not exist"
Sep 16 01:11:23.423982 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:23.423962    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9040-4992eb\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="b452b594-9f74-4e6b-9e80-69c32e53ca43" pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:23.425648 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:23.425630    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9043-e1b522\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:23.655649 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.655594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:23.812231 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.812190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"]
Sep 16 01:11:23.814659 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:23.814614    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9043-e1b522\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:23.837988 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:23.837939    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9040-4992eb\" is forbidden: User \"system:node:ip-10-0-130-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-4.ec2.internal' and this object" podUID="b452b594-9f74-4e6b-9e80-69c32e53ca43" pod="opendatahub/test-curl-9040-4992eb"
Sep 16 01:11:23.845374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.845341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:23.942682 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:23.942576    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b452b594-9f74-4e6b-9e80-69c32e53ca43" path="/var/lib/kubelet/pods/b452b594-9f74-4e6b-9e80-69c32e53ca43/volumes"
Sep 16 01:11:24.568566 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:24.568501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:24.825753 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:24.825650    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:25.286802 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.286768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:25.605015 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.604985    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:25.656564 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.656528    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnjqk\" (UniqueName: \"kubernetes.io/projected/ca0d8d5f-6283-4ac6-95c4-41a57db0cc79-kube-api-access-jnjqk\") pod \"ca0d8d5f-6283-4ac6-95c4-41a57db0cc79\" (UID: \"ca0d8d5f-6283-4ac6-95c4-41a57db0cc79\") "
Sep 16 01:11:25.658594 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.658561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0d8d5f-6283-4ac6-95c4-41a57db0cc79-kube-api-access-jnjqk" (OuterVolumeSpecName: "kube-api-access-jnjqk") pod "ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" (UID: "ca0d8d5f-6283-4ac6-95c4-41a57db0cc79"). InnerVolumeSpecName "kube-api-access-jnjqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:11:25.758023 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.757968    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnjqk\" (UniqueName: \"kubernetes.io/projected/ca0d8d5f-6283-4ac6-95c4-41a57db0cc79-kube-api-access-jnjqk\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:25.925374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.925276    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"]
Sep 16 01:11:25.925735 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.925717    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" containerName="test-curl-9043-e1b522"
Sep 16 01:11:25.925735 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.925734    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" containerName="test-curl-9043-e1b522"
Sep 16 01:11:25.925880 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.925832    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" containerName="test-curl-9043-e1b522"
Sep 16 01:11:25.928467 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.928194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:25.930611 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.930587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e7d6db19-serving-cert\""
Sep 16 01:11:25.944940 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.944912    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" path="/var/lib/kubelet/pods/ca0d8d5f-6283-4ac6-95c4-41a57db0cc79/volumes"
Sep 16 01:11:25.945423 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.945328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"]
Sep 16 01:11:25.957591 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.957563    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:25.959845 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.959822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2bn\" (UniqueName: \"kubernetes.io/projected/8054362c-51b5-484f-817a-8ae28eb85655-kube-api-access-nv2bn\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:25.959956 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.959852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:25.960016 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.959954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8054362c-51b5-484f-817a-8ae28eb85655-openshift-service-ca\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:25.960016 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:25.960002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-metrics-certs\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.058326 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.058287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:26.060534 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.060475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8054362c-51b5-484f-817a-8ae28eb85655-openshift-service-ca\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.060607 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.060586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-metrics-certs\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.060680 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.060652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2bn\" (UniqueName: \"kubernetes.io/projected/8054362c-51b5-484f-817a-8ae28eb85655-kube-api-access-nv2bn\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.060821 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.060757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.060914 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:26.060885    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-serving-cert: secret "maas-api-e2e-mt-e7d6db19-serving-cert" not found
Sep 16 01:11:26.060991 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:26.060973    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls podName:8054362c-51b5-484f-817a-8ae28eb85655 nodeName:}" failed. No retries permitted until 2026-09-16 01:11:26.560953799 +0000 UTC m=+1705.104090759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls") pod "maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" (UID: "8054362c-51b5-484f-817a-8ae28eb85655") : secret "maas-api-e2e-mt-e7d6db19-serving-cert" not found
Sep 16 01:11:26.061175 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.061154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8054362c-51b5-484f-817a-8ae28eb85655-openshift-service-ca\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.063056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.063038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-metrics-certs\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.070284 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.070251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2bn\" (UniqueName: \"kubernetes.io/projected/8054362c-51b5-484f-817a-8ae28eb85655-kube-api-access-nv2bn\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.419063 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.419026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:26.432861 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.432822    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca0d8d5f-6283-4ac6-95c4-41a57db0cc79" containerID="d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d" exitCode=137
Sep 16 01:11:26.433033 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.432994    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e1b522"
Sep 16 01:11:26.433103 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.432995    2581 scope.go:117] "RemoveContainer" containerID="d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d"
Sep 16 01:11:26.442959 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.442931    2581 scope.go:117] "RemoveContainer" containerID="d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d"
Sep 16 01:11:26.443270 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:26.443242    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d\": container with ID starting with d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d not found: ID does not exist" containerID="d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d"
Sep 16 01:11:26.443359 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.443276    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d"} err="failed to get container status \"d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d\": rpc error: code = NotFound desc = could not find container \"d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d\": container with ID starting with d97f43e8aa9b72c7d71cc9ca089e09684341424ede4af62a4083d22e68b2ee2d not found: ID does not exist"
Sep 16 01:11:26.501538 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.501503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:26.566030 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.565992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:26.566214 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:26.566167    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-serving-cert: secret "maas-api-e2e-mt-e7d6db19-serving-cert" not found
Sep 16 01:11:26.566291 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:26.566266    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls podName:8054362c-51b5-484f-817a-8ae28eb85655 nodeName:}" failed. No retries permitted until 2026-09-16 01:11:27.56624983 +0000 UTC m=+1706.109386782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls") pod "maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" (UID: "8054362c-51b5-484f-817a-8ae28eb85655") : secret "maas-api-e2e-mt-e7d6db19-serving-cert" not found
Sep 16 01:11:26.770649 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:26.770606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:27.128630 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:27.128540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:27.576408 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:27.576370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:27.579023 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:27.578987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls\") pod \"maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:27.744123 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:27.744069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:27.808042 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:27.807970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:28.016081 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:28.016054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"]
Sep 16 01:11:28.018375 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:11:28.018347    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8054362c_51b5_484f_817a_8ae28eb85655.slice/crio-c09c09fbc46998d04a9a3656ef88599e186c692be71f442c0f73f10e30722c91 WatchSource:0}: Error finding container c09c09fbc46998d04a9a3656ef88599e186c692be71f442c0f73f10e30722c91: Status 404 returned error can't find the container with id c09c09fbc46998d04a9a3656ef88599e186c692be71f442c0f73f10e30722c91
Sep 16 01:11:28.019701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:28.019677    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 01:11:28.446648 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:28.446615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" event={"ID":"8054362c-51b5-484f-817a-8ae28eb85655","Type":"ContainerStarted","Data":"87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7"}
Sep 16 01:11:28.446819 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:28.446676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" event={"ID":"8054362c-51b5-484f-817a-8ae28eb85655","Type":"ContainerStarted","Data":"c09c09fbc46998d04a9a3656ef88599e186c692be71f442c0f73f10e30722c91"}
Sep 16 01:11:28.446819 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:28.446724    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:28.463099 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:28.463042    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" podStartSLOduration=3.120068502 podStartE2EDuration="3.463026389s" totalImagesPullingTime="342.957887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:28.461247269 +0000 UTC m=+1707.004384268" watchObservedRunningTime="2026-09-16 01:11:28.463026389 +0000 UTC m=+1707.006163380"
Sep 16 01:11:28.807043 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:28.807005    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:32.189571 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.189537    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"]
Sep 16 01:11:32.193671 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.193650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.195882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.195849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f831c5\"/\"sub-test-model-f831c5-kserve-self-signed-certs\""
Sep 16 01:11:32.203062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.203033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"]
Sep 16 01:11:32.288542 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.288453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:32.323829 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.323777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tmp-dir\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.323829 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.323832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-home\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.324108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.323904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs57b\" (UniqueName: \"kubernetes.io/projected/c73fa298-e7d0-4a8a-a213-9ac31f203d44-kube-api-access-bs57b\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.324108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.323927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-dshm\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.324108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.324023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-model-cache\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.324108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.324046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tls-certs\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.424696 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.424656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tmp-dir\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.424696 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.424700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-home\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.424932 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.424848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs57b\" (UniqueName: \"kubernetes.io/projected/c73fa298-e7d0-4a8a-a213-9ac31f203d44-kube-api-access-bs57b\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.424932 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.424889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-dshm\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.425020 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.424984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-model-cache\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.425020 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.425000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tls-certs\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.425132 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.425092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-home\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.425194 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.425131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tmp-dir\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.425349 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.425326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-model-cache\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.427135 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.427112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-dshm\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.427438 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.427420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tls-certs\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.432032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.432002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs57b\" (UniqueName: \"kubernetes.io/projected/c73fa298-e7d0-4a8a-a213-9ac31f203d44-kube-api-access-bs57b\") pod \"sub-test-model-f831c5-kserve-549fcf6fd7-b78f7\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") " pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.504900 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.504799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:32.775982 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:32.775956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"]
Sep 16 01:11:32.777738 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:11:32.777709    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc73fa298_e7d0_4a8a_a213_9ac31f203d44.slice/crio-a3f25d3a692af01507445f58e4658396472ec92c8b55036f1869167903211ec3 WatchSource:0}: Error finding container a3f25d3a692af01507445f58e4658396472ec92c8b55036f1869167903211ec3: Status 404 returned error can't find the container with id a3f25d3a692af01507445f58e4658396472ec92c8b55036f1869167903211ec3
Sep 16 01:11:33.475153 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:33.475116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7" event={"ID":"c73fa298-e7d0-4a8a-a213-9ac31f203d44","Type":"ContainerStarted","Data":"2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f"}
Sep 16 01:11:33.475599 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:33.475162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7" event={"ID":"c73fa298-e7d0-4a8a-a213-9ac31f203d44","Type":"ContainerStarted","Data":"a3f25d3a692af01507445f58e4658396472ec92c8b55036f1869167903211ec3"}
Sep 16 01:11:33.494216 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:33.494154    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7" podStartSLOduration=1.49413795 podStartE2EDuration="1.49413795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:33.492041964 +0000 UTC m=+1712.035178938" watchObservedRunningTime="2026-09-16 01:11:33.49413795 +0000 UTC m=+1712.037274925"
Sep 16 01:11:34.094183 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:34.094143    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:11:34.141032 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:34.140994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:34.241437 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:34.241403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:34.458899 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:34.458822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:11:38.831792 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:38.831745    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5" podUID="3d1d22e3-bb0a-4a02-a52a-59ca07415e39" containerName="main" containerID="cri-o://8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633" gracePeriod=45
Sep 16 01:11:39.073829 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.073801    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:11:39.191287 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tmp-dir\") pod \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") "
Sep 16 01:11:39.191287 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191256    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tls-certs\") pod \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") "
Sep 16 01:11:39.191579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-dshm\") pod \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") "
Sep 16 01:11:39.191579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-model-cache\") pod \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") "
Sep 16 01:11:39.191579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-home\") pod \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") "
Sep 16 01:11:39.191579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d1d22e3-bb0a-4a02-a52a-59ca07415e39" (UID: "3d1d22e3-bb0a-4a02-a52a-59ca07415e39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:39.191579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191450    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-824gr\" (UniqueName: \"kubernetes.io/projected/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-kube-api-access-824gr\") pod \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\" (UID: \"3d1d22e3-bb0a-4a02-a52a-59ca07415e39\") "
Sep 16 01:11:39.191849 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d1d22e3-bb0a-4a02-a52a-59ca07415e39" (UID: "3d1d22e3-bb0a-4a02-a52a-59ca07415e39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:39.191849 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-home" (OuterVolumeSpecName: "home") pod "3d1d22e3-bb0a-4a02-a52a-59ca07415e39" (UID: "3d1d22e3-bb0a-4a02-a52a-59ca07415e39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:39.191930 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191898    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tmp-dir\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:39.191930 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191913    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-model-cache\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:39.191930 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.191923    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-home\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:39.193708 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.193683    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-kube-api-access-824gr" (OuterVolumeSpecName: "kube-api-access-824gr") pod "3d1d22e3-bb0a-4a02-a52a-59ca07415e39" (UID: "3d1d22e3-bb0a-4a02-a52a-59ca07415e39"). InnerVolumeSpecName "kube-api-access-824gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:11:39.193965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.193933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d1d22e3-bb0a-4a02-a52a-59ca07415e39" (UID: "3d1d22e3-bb0a-4a02-a52a-59ca07415e39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:11:39.194061 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.193933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-dshm" (OuterVolumeSpecName: "dshm") pod "3d1d22e3-bb0a-4a02-a52a-59ca07415e39" (UID: "3d1d22e3-bb0a-4a02-a52a-59ca07415e39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:39.292657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.292615    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824gr\" (UniqueName: \"kubernetes.io/projected/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-kube-api-access-824gr\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:39.292657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.292651    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-tls-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:39.292657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.292662    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1d22e3-bb0a-4a02-a52a-59ca07415e39-dshm\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:11:39.509051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.508962    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d1d22e3-bb0a-4a02-a52a-59ca07415e39" containerID="8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633" exitCode=0
Sep 16 01:11:39.509051 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.509008    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"
Sep 16 01:11:39.509242 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.509045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5" event={"ID":"3d1d22e3-bb0a-4a02-a52a-59ca07415e39","Type":"ContainerDied","Data":"8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633"}
Sep 16 01:11:39.509242 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.509094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5" event={"ID":"3d1d22e3-bb0a-4a02-a52a-59ca07415e39","Type":"ContainerDied","Data":"3fe312ec6cfae0784f1b86f8c6b448df543c5689cb732dc91af6e897e3eeadd1"}
Sep 16 01:11:39.509242 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.509116    2581 scope.go:117] "RemoveContainer" containerID="8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633"
Sep 16 01:11:39.519053 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.519027    2581 scope.go:117] "RemoveContainer" containerID="8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633"
Sep 16 01:11:39.519345 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:11:39.519322    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633\": container with ID starting with 8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633 not found: ID does not exist" containerID="8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633"
Sep 16 01:11:39.519432 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.519353    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633"} err="failed to get container status \"8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633\": rpc error: code = NotFound desc = could not find container \"8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633\": container with ID starting with 8f089059a19d80f89ace311a840de66fecd09784416f78f7b9dbf95186f5a633 not found: ID does not exist"
Sep 16 01:11:39.530395 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.530362    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"]
Sep 16 01:11:39.534417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.534393    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/auth-test-model-f831c5-kserve-ccf569b59-hzkl5"]
Sep 16 01:11:39.941398 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:39.941361    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1d22e3-bb0a-4a02-a52a-59ca07415e39" path="/var/lib/kubelet/pods/3d1d22e3-bb0a-4a02-a52a-59ca07415e39/volumes"
Sep 16 01:11:42.505138 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:42.505100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:42.505615 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:42.505177    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:42.518329 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:42.518297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:42.539719 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:42.539687    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:11:46.777890 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:46.777854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:47.609272 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:47.609234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:48.288837 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:48.288792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:48.403050 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:48.403021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:49.175681 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:49.175645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:50.003093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:50.003057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:50.780076 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:50.780040    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:51.193467 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:51.193371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:11:51.608010 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:51.607973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:53.904706 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:53.904670    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:56.108630 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:56.108584    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:56.161955 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:56.161923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:59.857061 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:59.857021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:11:59.920418 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:11:59.920382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:01.984786 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:01.984741    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:12:07.216716 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:07.216673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:15.390683 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:15.390638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:12:35.181921 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:35.181882    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:12:38.772401 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:38.772342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:39.493170 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:39.493135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:40.211691 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:40.211627    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:41.120240 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:41.120195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:41.791139 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:41.791110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:42.551347 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:42.551314    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:50.581814 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:50.581769    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:12:54.022219 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.022132    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"]
Sep 16 01:12:54.022737 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.022394    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" podUID="8054362c-51b5-484f-817a-8ae28eb85655" containerName="maas-api" containerID="cri-o://87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7" gracePeriod=30
Sep 16 01:12:54.051260 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.051212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:54.151802 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.151764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:54.259374 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.259342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:54.278301 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.278244    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:12:54.344238 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.344207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:54.404503 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.404445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8054362c-51b5-484f-817a-8ae28eb85655-openshift-service-ca\") pod \"8054362c-51b5-484f-817a-8ae28eb85655\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") "
Sep 16 01:12:54.404700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.404548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls\") pod \"8054362c-51b5-484f-817a-8ae28eb85655\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") "
Sep 16 01:12:54.404700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.404583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv2bn\" (UniqueName: \"kubernetes.io/projected/8054362c-51b5-484f-817a-8ae28eb85655-kube-api-access-nv2bn\") pod \"8054362c-51b5-484f-817a-8ae28eb85655\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") "
Sep 16 01:12:54.404700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.404640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-metrics-certs\") pod \"8054362c-51b5-484f-817a-8ae28eb85655\" (UID: \"8054362c-51b5-484f-817a-8ae28eb85655\") "
Sep 16 01:12:54.404944 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.404915    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8054362c-51b5-484f-817a-8ae28eb85655-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8054362c-51b5-484f-817a-8ae28eb85655" (UID: "8054362c-51b5-484f-817a-8ae28eb85655"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:12:54.406858 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.406823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8054362c-51b5-484f-817a-8ae28eb85655" (UID: "8054362c-51b5-484f-817a-8ae28eb85655"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:12:54.406858 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.406850    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8054362c-51b5-484f-817a-8ae28eb85655" (UID: "8054362c-51b5-484f-817a-8ae28eb85655"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:12:54.406998 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.406880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8054362c-51b5-484f-817a-8ae28eb85655-kube-api-access-nv2bn" (OuterVolumeSpecName: "kube-api-access-nv2bn") pod "8054362c-51b5-484f-817a-8ae28eb85655" (UID: "8054362c-51b5-484f-817a-8ae28eb85655"). InnerVolumeSpecName "kube-api-access-nv2bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:12:54.505563 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.505519    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8054362c-51b5-484f-817a-8ae28eb85655-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:12:54.505563 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.505562    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:12:54.505788 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.505574    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2bn\" (UniqueName: \"kubernetes.io/projected/8054362c-51b5-484f-817a-8ae28eb85655-kube-api-access-nv2bn\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:12:54.505788 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.505584    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8054362c-51b5-484f-817a-8ae28eb85655-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:12:54.702447 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.702353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:54.913882 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.913846    2581 generic.go:356] "Generic (PLEG): container finished" podID="8054362c-51b5-484f-817a-8ae28eb85655" containerID="87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7" exitCode=0
Sep 16 01:12:54.914075 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.913899    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"
Sep 16 01:12:54.914075 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.913930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" event={"ID":"8054362c-51b5-484f-817a-8ae28eb85655","Type":"ContainerDied","Data":"87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7"}
Sep 16 01:12:54.914075 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.914007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk" event={"ID":"8054362c-51b5-484f-817a-8ae28eb85655","Type":"ContainerDied","Data":"c09c09fbc46998d04a9a3656ef88599e186c692be71f442c0f73f10e30722c91"}
Sep 16 01:12:54.914075 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.914030    2581 scope.go:117] "RemoveContainer" containerID="87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7"
Sep 16 01:12:54.924331 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.924312    2581 scope.go:117] "RemoveContainer" containerID="87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7"
Sep 16 01:12:54.924712 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:12:54.924692    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7\": container with ID starting with 87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7 not found: ID does not exist" containerID="87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7"
Sep 16 01:12:54.924762 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.924722    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7"} err="failed to get container status \"87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7\": rpc error: code = NotFound desc = could not find container \"87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7\": container with ID starting with 87f0d591b09f9a099590376c21d3b124d4470c8a1e85576c9f4b44e8b023b8f7 not found: ID does not exist"
Sep 16 01:12:54.936061 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.936029    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"]
Sep 16 01:12:54.941999 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:54.941966    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7d6db19-7fff68f565-w96xk"]
Sep 16 01:12:55.495281 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:55.495246    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:55.941238 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:55.941206    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8054362c-51b5-484f-817a-8ae28eb85655" path="/var/lib/kubelet/pods/8054362c-51b5-484f-817a-8ae28eb85655/volumes"
Sep 16 01:12:56.314038 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:56.314006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:56.957659 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:56.957620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:12:57.587029 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:12:57.586996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:02.009859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:02.009833    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:13:02.017926 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:02.017898    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:13:02.610119 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:02.610064    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:03.628531 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.628497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9043-e76dfe"]
Sep 16 01:13:03.628932 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.628918    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8054362c-51b5-484f-817a-8ae28eb85655" containerName="maas-api"
Sep 16 01:13:03.628981 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.628934    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8054362c-51b5-484f-817a-8ae28eb85655" containerName="maas-api"
Sep 16 01:13:03.628981 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.628953    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1d22e3-bb0a-4a02-a52a-59ca07415e39" containerName="main"
Sep 16 01:13:03.628981 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.628959    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1d22e3-bb0a-4a02-a52a-59ca07415e39" containerName="main"
Sep 16 01:13:03.629078 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.629020    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8054362c-51b5-484f-817a-8ae28eb85655" containerName="maas-api"
Sep 16 01:13:03.629078 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.629033    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1d22e3-bb0a-4a02-a52a-59ca07415e39" containerName="main"
Sep 16 01:13:03.633103 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.633081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e76dfe"
Sep 16 01:13:03.635431 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.635407    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c6q92\""
Sep 16 01:13:03.636379 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.636353    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9043-e76dfe"]
Sep 16 01:13:03.695948 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.695916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kbsc\" (UniqueName: \"kubernetes.io/projected/57940884-b57c-473c-b063-5c46615aa8ff-kube-api-access-2kbsc\") pod \"test-curl-9043-e76dfe\" (UID: \"57940884-b57c-473c-b063-5c46615aa8ff\") " pod="opendatahub/test-curl-9043-e76dfe"
Sep 16 01:13:03.797109 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.797072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kbsc\" (UniqueName: \"kubernetes.io/projected/57940884-b57c-473c-b063-5c46615aa8ff-kube-api-access-2kbsc\") pod \"test-curl-9043-e76dfe\" (UID: \"57940884-b57c-473c-b063-5c46615aa8ff\") " pod="opendatahub/test-curl-9043-e76dfe"
Sep 16 01:13:03.805764 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.805741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kbsc\" (UniqueName: \"kubernetes.io/projected/57940884-b57c-473c-b063-5c46615aa8ff-kube-api-access-2kbsc\") pod \"test-curl-9043-e76dfe\" (UID: \"57940884-b57c-473c-b063-5c46615aa8ff\") " pod="opendatahub/test-curl-9043-e76dfe"
Sep 16 01:13:03.943499 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:03.943400    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e76dfe"
Sep 16 01:13:04.223050 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:04.223023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9043-e76dfe"]
Sep 16 01:13:04.224725 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:13:04.224690    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57940884_b57c_473c_b063_5c46615aa8ff.slice/crio-87f4ace618bd48194bc14f70f84868e8898f4e6223ccc9d44bd9713292984408 WatchSource:0}: Error finding container 87f4ace618bd48194bc14f70f84868e8898f4e6223ccc9d44bd9713292984408: Status 404 returned error can't find the container with id 87f4ace618bd48194bc14f70f84868e8898f4e6223ccc9d44bd9713292984408
Sep 16 01:13:04.969562 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:04.969517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9043-e76dfe" event={"ID":"57940884-b57c-473c-b063-5c46615aa8ff","Type":"ContainerStarted","Data":"e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea"}
Sep 16 01:13:04.969562 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:04.969558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9043-e76dfe" event={"ID":"57940884-b57c-473c-b063-5c46615aa8ff","Type":"ContainerStarted","Data":"87f4ace618bd48194bc14f70f84868e8898f4e6223ccc9d44bd9713292984408"}
Sep 16 01:13:04.983395 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:04.983345    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9043-e76dfe" podStartSLOduration=1.9833290940000001 podStartE2EDuration="1.983329094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:13:04.981620287 +0000 UTC m=+1803.524757261" watchObservedRunningTime="2026-09-16 01:13:04.983329094 +0000 UTC m=+1803.526466069"
Sep 16 01:13:05.551430 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.551390    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9043-e76dfe"]
Sep 16 01:13:05.555164 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.555131    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9043-e76dfe"]
Sep 16 01:13:05.672916 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.672830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9043-dd8947"]
Sep 16 01:13:05.673251 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.673236    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57940884-b57c-473c-b063-5c46615aa8ff" containerName="test-curl-9043-e76dfe"
Sep 16 01:13:05.673295 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.673255    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57940884-b57c-473c-b063-5c46615aa8ff" containerName="test-curl-9043-e76dfe"
Sep 16 01:13:05.673348 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.673338    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57940884-b57c-473c-b063-5c46615aa8ff" containerName="test-curl-9043-e76dfe"
Sep 16 01:13:05.676342 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.676326    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-dd8947"
Sep 16 01:13:05.681828 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.681798    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9043-dd8947"]
Sep 16 01:13:05.817570 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.817537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnvf2\" (UniqueName: \"kubernetes.io/projected/34497006-72c1-47f3-ad4e-6235adbfc9b5-kube-api-access-tnvf2\") pod \"test-curl-9043-dd8947\" (UID: \"34497006-72c1-47f3-ad4e-6235adbfc9b5\") " pod="opendatahub/test-curl-9043-dd8947"
Sep 16 01:13:05.918812 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.918774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnvf2\" (UniqueName: \"kubernetes.io/projected/34497006-72c1-47f3-ad4e-6235adbfc9b5-kube-api-access-tnvf2\") pod \"test-curl-9043-dd8947\" (UID: \"34497006-72c1-47f3-ad4e-6235adbfc9b5\") " pod="opendatahub/test-curl-9043-dd8947"
Sep 16 01:13:05.926518 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.926432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnvf2\" (UniqueName: \"kubernetes.io/projected/34497006-72c1-47f3-ad4e-6235adbfc9b5-kube-api-access-tnvf2\") pod \"test-curl-9043-dd8947\" (UID: \"34497006-72c1-47f3-ad4e-6235adbfc9b5\") " pod="opendatahub/test-curl-9043-dd8947"
Sep 16 01:13:05.974561 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:13:05.974531    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9043-e76dfe" container="test-curl-9043-e76dfe"
Sep 16 01:13:05.987445 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:05.987418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-dd8947"
Sep 16 01:13:06.257009 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:06.256983    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9043-dd8947"]
Sep 16 01:13:06.258710 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:13:06.258683    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34497006_72c1_47f3_ad4e_6235adbfc9b5.slice/crio-124e79fc967390a1130cc2a0402b3b2fe8e897c99070606a8ce2cdc13be453dc WatchSource:0}: Error finding container 124e79fc967390a1130cc2a0402b3b2fe8e897c99070606a8ce2cdc13be453dc: Status 404 returned error can't find the container with id 124e79fc967390a1130cc2a0402b3b2fe8e897c99070606a8ce2cdc13be453dc
Sep 16 01:13:06.981902 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:06.981855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9043-dd8947" event={"ID":"34497006-72c1-47f3-ad4e-6235adbfc9b5","Type":"ContainerStarted","Data":"09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c"}
Sep 16 01:13:06.981902 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:06.981906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9043-dd8947" event={"ID":"34497006-72c1-47f3-ad4e-6235adbfc9b5","Type":"ContainerStarted","Data":"124e79fc967390a1130cc2a0402b3b2fe8e897c99070606a8ce2cdc13be453dc"}
Sep 16 01:13:06.982411 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:06.981949    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9043-e76dfe" podUID="57940884-b57c-473c-b063-5c46615aa8ff" containerName="test-curl-9043-e76dfe" containerID="cri-o://e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea" gracePeriod=2
Sep 16 01:13:06.995793 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:06.995743    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9043-dd8947" podStartSLOduration=1.995727691 podStartE2EDuration="1.995727691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:13:06.993807396 +0000 UTC m=+1805.536944372" watchObservedRunningTime="2026-09-16 01:13:06.995727691 +0000 UTC m=+1805.538864665"
Sep 16 01:13:07.570887 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:07.570847    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9043-dd8947"]
Sep 16 01:13:07.577324 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:07.577291    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9043-dd8947"]
Sep 16 01:13:07.986768 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:13:07.986738    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9043-dd8947" container="test-curl-9043-dd8947"
Sep 16 01:13:08.991521 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:08.991451    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9043-dd8947" podUID="34497006-72c1-47f3-ad4e-6235adbfc9b5" containerName="test-curl-9043-dd8947" containerID="cri-o://09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c" gracePeriod=2
Sep 16 01:13:09.158218 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.158192    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e76dfe"
Sep 16 01:13:09.247309 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.247219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kbsc\" (UniqueName: \"kubernetes.io/projected/57940884-b57c-473c-b063-5c46615aa8ff-kube-api-access-2kbsc\") pod \"57940884-b57c-473c-b063-5c46615aa8ff\" (UID: \"57940884-b57c-473c-b063-5c46615aa8ff\") "
Sep 16 01:13:09.249288 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.249258    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57940884-b57c-473c-b063-5c46615aa8ff-kube-api-access-2kbsc" (OuterVolumeSpecName: "kube-api-access-2kbsc") pod "57940884-b57c-473c-b063-5c46615aa8ff" (UID: "57940884-b57c-473c-b063-5c46615aa8ff"). InnerVolumeSpecName "kube-api-access-2kbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:13:09.348748 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.348709    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kbsc\" (UniqueName: \"kubernetes.io/projected/57940884-b57c-473c-b063-5c46615aa8ff-kube-api-access-2kbsc\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:09.587135 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.587100    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"]
Sep 16 01:13:09.605838 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.605803    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:09.941884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.941801    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57940884-b57c-473c-b063-5c46615aa8ff" path="/var/lib/kubelet/pods/57940884-b57c-473c-b063-5c46615aa8ff/volumes"
Sep 16 01:13:09.997555 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.997520    2581 generic.go:356] "Generic (PLEG): container finished" podID="57940884-b57c-473c-b063-5c46615aa8ff" containerID="e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea" exitCode=137
Sep 16 01:13:09.998041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.997585    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-e76dfe"
Sep 16 01:13:09.998041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:09.997629    2581 scope.go:117] "RemoveContainer" containerID="e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea"
Sep 16 01:13:10.006062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:10.006041    2581 scope.go:117] "RemoveContainer" containerID="e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea"
Sep 16 01:13:10.006365 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:13:10.006345    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea\": container with ID starting with e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea not found: ID does not exist" containerID="e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea"
Sep 16 01:13:10.006409 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:10.006374    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea"} err="failed to get container status \"e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea\": rpc error: code = NotFound desc = could not find container \"e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea\": container with ID starting with e442937ad849f9f0fcf9bdccf66b885a29723872b177609be2441251e5a5f4ea not found: ID does not exist"
Sep 16 01:13:10.718170 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:10.718082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:11.167111 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:11.167079    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-dd8947"
Sep 16 01:13:11.268280 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:11.268251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnvf2\" (UniqueName: \"kubernetes.io/projected/34497006-72c1-47f3-ad4e-6235adbfc9b5-kube-api-access-tnvf2\") pod \"34497006-72c1-47f3-ad4e-6235adbfc9b5\" (UID: \"34497006-72c1-47f3-ad4e-6235adbfc9b5\") "
Sep 16 01:13:11.270419 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:11.270392    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34497006-72c1-47f3-ad4e-6235adbfc9b5-kube-api-access-tnvf2" (OuterVolumeSpecName: "kube-api-access-tnvf2") pod "34497006-72c1-47f3-ad4e-6235adbfc9b5" (UID: "34497006-72c1-47f3-ad4e-6235adbfc9b5"). InnerVolumeSpecName "kube-api-access-tnvf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:13:11.369392 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:11.369353    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnvf2\" (UniqueName: \"kubernetes.io/projected/34497006-72c1-47f3-ad4e-6235adbfc9b5-kube-api-access-tnvf2\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:11.943108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:11.943065    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34497006-72c1-47f3-ad4e-6235adbfc9b5" path="/var/lib/kubelet/pods/34497006-72c1-47f3-ad4e-6235adbfc9b5/volumes"
Sep 16 01:13:12.009811 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:12.009772    2581 generic.go:356] "Generic (PLEG): container finished" podID="34497006-72c1-47f3-ad4e-6235adbfc9b5" containerID="09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c" exitCode=137
Sep 16 01:13:12.009811 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:12.009814    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9043-dd8947"
Sep 16 01:13:12.010054 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:12.009866    2581 scope.go:117] "RemoveContainer" containerID="09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c"
Sep 16 01:13:12.018456 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:12.018437    2581 scope.go:117] "RemoveContainer" containerID="09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c"
Sep 16 01:13:12.018781 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:13:12.018759    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c\": container with ID starting with 09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c not found: ID does not exist" containerID="09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c"
Sep 16 01:13:12.018872 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:12.018793    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c"} err="failed to get container status \"09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c\": rpc error: code = NotFound desc = could not find container \"09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c\": container with ID starting with 09985e2ac3bdd1e1f1980d3d242995b79b8636c2b886f97c143314375b106e8c not found: ID does not exist"
Sep 16 01:13:12.482242 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:12.482203    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:13:17.030761 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:17.030719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:24.606874 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.606827    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7" podUID="c73fa298-e7d0-4a8a-a213-9ac31f203d44" containerName="main" containerID="cri-o://2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f" gracePeriod=45
Sep 16 01:13:24.864415 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.864355    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:13:24.904380 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-model-cache\") pod \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") "
Sep 16 01:13:24.904575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-dshm\") pod \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") "
Sep 16 01:13:24.904575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tls-certs\") pod \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") "
Sep 16 01:13:24.904575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904472    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs57b\" (UniqueName: \"kubernetes.io/projected/c73fa298-e7d0-4a8a-a213-9ac31f203d44-kube-api-access-bs57b\") pod \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") "
Sep 16 01:13:24.904575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tmp-dir\") pod \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") "
Sep 16 01:13:24.904575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-home\") pod \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\" (UID: \"c73fa298-e7d0-4a8a-a213-9ac31f203d44\") "
Sep 16 01:13:24.904890 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-model-cache" (OuterVolumeSpecName: "model-cache") pod "c73fa298-e7d0-4a8a-a213-9ac31f203d44" (UID: "c73fa298-e7d0-4a8a-a213-9ac31f203d44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:24.904890 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c73fa298-e7d0-4a8a-a213-9ac31f203d44" (UID: "c73fa298-e7d0-4a8a-a213-9ac31f203d44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:24.905010 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904899    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-home" (OuterVolumeSpecName: "home") pod "c73fa298-e7d0-4a8a-a213-9ac31f203d44" (UID: "c73fa298-e7d0-4a8a-a213-9ac31f203d44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:24.905010 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.904926    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-model-cache\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:24.906672 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.906641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-dshm" (OuterVolumeSpecName: "dshm") pod "c73fa298-e7d0-4a8a-a213-9ac31f203d44" (UID: "c73fa298-e7d0-4a8a-a213-9ac31f203d44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:24.906802 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.906737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c73fa298-e7d0-4a8a-a213-9ac31f203d44" (UID: "c73fa298-e7d0-4a8a-a213-9ac31f203d44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:13:24.906873 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:24.906809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73fa298-e7d0-4a8a-a213-9ac31f203d44-kube-api-access-bs57b" (OuterVolumeSpecName: "kube-api-access-bs57b") pod "c73fa298-e7d0-4a8a-a213-9ac31f203d44" (UID: "c73fa298-e7d0-4a8a-a213-9ac31f203d44"). InnerVolumeSpecName "kube-api-access-bs57b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:13:25.006316 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.006202    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-dshm\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:25.006316 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.006236    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tls-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:25.006316 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.006251    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs57b\" (UniqueName: \"kubernetes.io/projected/c73fa298-e7d0-4a8a-a213-9ac31f203d44-kube-api-access-bs57b\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:25.006316 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.006267    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-tmp-dir\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:25.006316 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.006282    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73fa298-e7d0-4a8a-a213-9ac31f203d44-home\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:13:25.085773 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.085736    2581 generic.go:356] "Generic (PLEG): container finished" podID="c73fa298-e7d0-4a8a-a213-9ac31f203d44" containerID="2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f" exitCode=0
Sep 16 01:13:25.085951 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.085786    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"
Sep 16 01:13:25.085951 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.085815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7" event={"ID":"c73fa298-e7d0-4a8a-a213-9ac31f203d44","Type":"ContainerDied","Data":"2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f"}
Sep 16 01:13:25.085951 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.085858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7" event={"ID":"c73fa298-e7d0-4a8a-a213-9ac31f203d44","Type":"ContainerDied","Data":"a3f25d3a692af01507445f58e4658396472ec92c8b55036f1869167903211ec3"}
Sep 16 01:13:25.085951 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.085878    2581 scope.go:117] "RemoveContainer" containerID="2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f"
Sep 16 01:13:25.095748 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.095725    2581 scope.go:117] "RemoveContainer" containerID="2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f"
Sep 16 01:13:25.096051 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:13:25.096031    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f\": container with ID starting with 2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f not found: ID does not exist" containerID="2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f"
Sep 16 01:13:25.096114 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.096059    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f"} err="failed to get container status \"2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f\": rpc error: code = NotFound desc = could not find container \"2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f\": container with ID starting with 2d6b84feaa0a1a029aa2fc244e1818508326efd499a6e672070022fcae50291f not found: ID does not exist"
Sep 16 01:13:25.106880 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.106853    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"]
Sep 16 01:13:25.114276 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.114233    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f831c5/sub-test-model-f831c5-kserve-549fcf6fd7-b78f7"]
Sep 16 01:13:25.941472 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:25.941422    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73fa298-e7d0-4a8a-a213-9ac31f203d44" path="/var/lib/kubelet/pods/c73fa298-e7d0-4a8a-a213-9ac31f203d44/volumes"
Sep 16 01:13:28.380678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:28.380638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:13:39.267574 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:39.267537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:40.505868 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:40.505831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:45.221746 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:45.221709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:46.681195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.681149    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"]
Sep 16 01:13:46.681698 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.681678    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34497006-72c1-47f3-ad4e-6235adbfc9b5" containerName="test-curl-9043-dd8947"
Sep 16 01:13:46.681745 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.681700    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34497006-72c1-47f3-ad4e-6235adbfc9b5" containerName="test-curl-9043-dd8947"
Sep 16 01:13:46.681745 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.681733    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73fa298-e7d0-4a8a-a213-9ac31f203d44" containerName="main"
Sep 16 01:13:46.681745 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.681742    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73fa298-e7d0-4a8a-a213-9ac31f203d44" containerName="main"
Sep 16 01:13:46.681873 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.681860    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34497006-72c1-47f3-ad4e-6235adbfc9b5" containerName="test-curl-9043-dd8947"
Sep 16 01:13:46.681910 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.681875    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73fa298-e7d0-4a8a-a213-9ac31f203d44" containerName="main"
Sep 16 01:13:46.685131 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.685107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.688600 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.688569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"default-dockercfg-rnjz5\""
Sep 16 01:13:46.688600 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.688591    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"kube-root-ca.crt\""
Sep 16 01:13:46.688812 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.688604    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"rate-test-model-0ee1bf-kserve-self-signed-certs\""
Sep 16 01:13:46.688812 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.688569    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"openshift-service-ca.crt\""
Sep 16 01:13:46.694817 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.694790    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"]
Sep 16 01:13:46.710358 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.710326    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:46.803782 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.803738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-tmp-dir\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.803977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.803820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-home\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.803977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.803855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd2b445-684a-4473-81a5-5caf97c818e4-tls-certs\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.803977 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.803940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-model-cache\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.804117 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.803976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkvtz\" (UniqueName: \"kubernetes.io/projected/7bd2b445-684a-4473-81a5-5caf97c818e4-kube-api-access-qkvtz\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.804117 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.804000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-dshm\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.904566 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.904531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-tmp-dir\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.904800 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.904607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-home\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.904800 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.904631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd2b445-684a-4473-81a5-5caf97c818e4-tls-certs\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.904800 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.904679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-model-cache\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.904800 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.904703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkvtz\" (UniqueName: \"kubernetes.io/projected/7bd2b445-684a-4473-81a5-5caf97c818e4-kube-api-access-qkvtz\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.904800 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.904722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-dshm\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.905082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.905002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-tmp-dir\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.905082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.905023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-home\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.905082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.905063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-model-cache\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.907138 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.907111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-dshm\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.907346 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.907330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd2b445-684a-4473-81a5-5caf97c818e4-tls-certs\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.912261 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.912239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkvtz\" (UniqueName: \"kubernetes.io/projected/7bd2b445-684a-4473-81a5-5caf97c818e4-kube-api-access-qkvtz\") pod \"rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:46.996122 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:46.996036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:47.269649 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:47.269620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"]
Sep 16 01:13:47.271187 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:13:47.271162    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd2b445_684a_4473_81a5_5caf97c818e4.slice/crio-a84c16c0b91c0f9768576f477d4a8260c361feb7607cb1e597ec76575ca10947 WatchSource:0}: Error finding container a84c16c0b91c0f9768576f477d4a8260c361feb7607cb1e597ec76575ca10947: Status 404 returned error can't find the container with id a84c16c0b91c0f9768576f477d4a8260c361feb7607cb1e597ec76575ca10947
Sep 16 01:13:48.219044 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:48.219000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj" event={"ID":"7bd2b445-684a-4473-81a5-5caf97c818e4","Type":"ContainerStarted","Data":"d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0"}
Sep 16 01:13:48.219452 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:48.219056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj" event={"ID":"7bd2b445-684a-4473-81a5-5caf97c818e4","Type":"ContainerStarted","Data":"a84c16c0b91c0f9768576f477d4a8260c361feb7607cb1e597ec76575ca10947"}
Sep 16 01:13:48.235376 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:48.235297    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj" podStartSLOduration=2.235281525 podStartE2EDuration="2.235281525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:13:48.234360696 +0000 UTC m=+1846.777497665" watchObservedRunningTime="2026-09-16 01:13:48.235281525 +0000 UTC m=+1846.778418501"
Sep 16 01:13:51.166571 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:51.166531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:51.264527 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:51.264472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:51.298379 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:51.298340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:13:51.428097 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:51.428008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:52.765721 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:52.765682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:52.868141 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:52.868110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:52.960626 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:52.960586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:53.603218 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:53.603183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:53.704732 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:53.704698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:53.802202 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:53.802151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:55.070846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:55.070811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:13:56.996190 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:56.996156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:56.996610 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:56.996213    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:57.009282 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:57.009254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:13:57.281715 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:13:57.281684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:14:00.853775 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:00.853739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:00.913403 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:00.913370    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:01.572293 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:01.572258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:07.171450 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:07.171412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:07.272526 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:07.272471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:07.367980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:07.367942    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:08.484702 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:08.484663    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:14:17.904515 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:17.904453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:18.141121 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:18.141069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:27.478244 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:27.478155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:14:33.208074 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:33.208039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:34.196281 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:34.196250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"]
Sep 16 01:14:34.215198 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:34.215163    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:34.686066 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:34.686036    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:35.003548 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:35.003432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:35.147629 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:35.147592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:38.225814 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:38.225782    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:38.497398 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:38.497311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:38.595699 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:38.595664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:39.072476 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:39.072441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:39.218559 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:39.218513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:39.999377 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:39.999340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:40.692304 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:40.692268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:14:40.694652 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:40.694620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:41.478125 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.478081    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"]
Sep 16 01:14:41.481379 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.481351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.483413 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.483393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-089563-serving-cert\""
Sep 16 01:14:41.494069 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.494044    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"]
Sep 16 01:14:41.510449 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.510421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:41.565458 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.565428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:41.620129 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.620091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.620338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.620154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdhn\" (UniqueName: \"kubernetes.io/projected/53031bfb-5aa6-41b1-a856-21781e87c274-kube-api-access-nfdhn\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.620338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.620249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-metrics-certs\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.620338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.620289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53031bfb-5aa6-41b1-a856-21781e87c274-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.721150 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.721112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdhn\" (UniqueName: \"kubernetes.io/projected/53031bfb-5aa6-41b1-a856-21781e87c274-kube-api-access-nfdhn\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.721360 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.721164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-metrics-certs\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.721360 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.721192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53031bfb-5aa6-41b1-a856-21781e87c274-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.721360 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.721281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.721591 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:14:41.721384    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-serving-cert: secret "maas-api-e2e-ipp-a-089563-serving-cert" not found
Sep 16 01:14:41.721591 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:14:41.721447    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls podName:53031bfb-5aa6-41b1-a856-21781e87c274 nodeName:}" failed. No retries permitted until 2026-09-16 01:14:42.221428758 +0000 UTC m=+1900.764565736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls") pod "maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v" (UID: "53031bfb-5aa6-41b1-a856-21781e87c274") : secret "maas-api-e2e-ipp-a-089563-serving-cert" not found
Sep 16 01:14:41.722038 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.722011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53031bfb-5aa6-41b1-a856-21781e87c274-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.723638 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.723617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-metrics-certs\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.729365 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.729305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdhn\" (UniqueName: \"kubernetes.io/projected/53031bfb-5aa6-41b1-a856-21781e87c274-kube-api-access-nfdhn\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:41.783164 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.783130    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:41.957557 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:41.957528    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:42.064290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:42.064263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:42.226992 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:42.226944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:42.229422 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:42.229399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls\") pod \"maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:42.394565 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:42.394449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:42.670861 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:42.670832    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"]
Sep 16 01:14:42.673709 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:14:42.673682    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53031bfb_5aa6_41b1_a856_21781e87c274.slice/crio-44cf716064336c5afa7b00aaa1900260f53506722e302ee3004708de478555e3 WatchSource:0}: Error finding container 44cf716064336c5afa7b00aaa1900260f53506722e302ee3004708de478555e3: Status 404 returned error can't find the container with id 44cf716064336c5afa7b00aaa1900260f53506722e302ee3004708de478555e3
Sep 16 01:14:43.512338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:43.512303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v" event={"ID":"53031bfb-5aa6-41b1-a856-21781e87c274","Type":"ContainerStarted","Data":"9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96"}
Sep 16 01:14:43.512529 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:43.512365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:43.512529 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:43.512382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v" event={"ID":"53031bfb-5aa6-41b1-a856-21781e87c274","Type":"ContainerStarted","Data":"44cf716064336c5afa7b00aaa1900260f53506722e302ee3004708de478555e3"}
Sep 16 01:14:43.527980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:43.527930    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v" podStartSLOduration=2.155933898 podStartE2EDuration="2.527913505s" totalImagesPullingTime="371.979607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:43.527253586 +0000 UTC m=+1902.070390565" watchObservedRunningTime="2026-09-16 01:14:43.527913505 +0000 UTC m=+1902.071050480"
Sep 16 01:14:44.969902 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:44.969863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:45.034798 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.034758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:45.304497 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.304449    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:45.579755 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.579659    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"]
Sep 16 01:14:45.583600 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.583575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.586168 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.586145    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ad13e5-openshift-default-dockercfg-v9c2z\""
Sep 16 01:14:45.594643 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.594616    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"]
Sep 16 01:14:45.612263 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.612228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:45.759328 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-certs\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759328 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-envoy\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759581 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-data\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759581 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759436    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-token\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759581 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-credential-socket\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759581 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-podinfo\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759581 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-socket\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759746 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d077d476-a1e1-4476-99b3-f8bc4e431eea-istiod-ca-cert\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.759746 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.759693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtk6p\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-kube-api-access-vtk6p\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.860942 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.860835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-credential-socket\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.860942 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.860899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-podinfo\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.860942 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.860926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-socket\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861239 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.860951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d077d476-a1e1-4476-99b3-f8bc4e431eea-istiod-ca-cert\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861239 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.860981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtk6p\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-kube-api-access-vtk6p\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861239 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-certs\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861239 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-envoy\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861239 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-data\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861239 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-token\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861553 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-socket\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861623 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-data\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861679 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-credential-socket\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.861850 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.861831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d077d476-a1e1-4476-99b3-f8bc4e431eea-istiod-ca-cert\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.862071 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.862052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-certs\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.863497 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.863463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-envoy\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.863686 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.863667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-podinfo\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.869170 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.869144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-token\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.869366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.869346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtk6p\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-kube-api-access-vtk6p\") pod \"e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") " pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:45.896359 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:45.896321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:46.025577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.025542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:46.174598 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.174569    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"]
Sep 16 01:14:46.176574 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:14:46.176538    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd077d476_a1e1_4476_99b3_f8bc4e431eea.slice/crio-65c358b97fce7f0f6a11aa003c8e7a7b9588cf4985d7d63e2f4e3792b5c7c38e WatchSource:0}: Error finding container 65c358b97fce7f0f6a11aa003c8e7a7b9588cf4985d7d63e2f4e3792b5c7c38e: Status 404 returned error can't find the container with id 65c358b97fce7f0f6a11aa003c8e7a7b9588cf4985d7d63e2f4e3792b5c7c38e
Sep 16 01:14:46.178880 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.178848    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:14:46.178976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.178914    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:14:46.178976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.178942    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:14:46.532417 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.532384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" event={"ID":"d077d476-a1e1-4476-99b3-f8bc4e431eea","Type":"ContainerStarted","Data":"8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a"}
Sep 16 01:14:46.532722 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.532448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" event={"ID":"d077d476-a1e1-4476-99b3-f8bc4e431eea","Type":"ContainerStarted","Data":"65c358b97fce7f0f6a11aa003c8e7a7b9588cf4985d7d63e2f4e3792b5c7c38e"}
Sep 16 01:14:46.550631 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.550573    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podStartSLOduration=1.5505563489999998 podStartE2EDuration="1.550556349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:46.548189686 +0000 UTC m=+1905.091326653" watchObservedRunningTime="2026-09-16 01:14:46.550556349 +0000 UTC m=+1905.093693390"
Sep 16 01:14:46.896956 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:46.896864    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:47.720412 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:47.720377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:47.902664 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:47.902634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:48.492499 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:48.492442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:48.544049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:48.544015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:48.545047 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:48.545025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:14:48.590602 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:48.590557    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:48.690503 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:48.690448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:48.793093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:48.793060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:49.216364 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.216274    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj" podUID="7bd2b445-684a-4473-81a5-5caf97c818e4" containerName="main" containerID="cri-o://d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0" gracePeriod=45
Sep 16 01:14:49.476996 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.476929    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:14:49.523084 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.523052    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:14:49.564903 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.564868    2581 generic.go:356] "Generic (PLEG): container finished" podID="7bd2b445-684a-4473-81a5-5caf97c818e4" containerID="d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0" exitCode=0
Sep 16 01:14:49.565095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.564930    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"
Sep 16 01:14:49.565095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.564950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj" event={"ID":"7bd2b445-684a-4473-81a5-5caf97c818e4","Type":"ContainerDied","Data":"d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0"}
Sep 16 01:14:49.565095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.564997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj" event={"ID":"7bd2b445-684a-4473-81a5-5caf97c818e4","Type":"ContainerDied","Data":"a84c16c0b91c0f9768576f477d4a8260c361feb7607cb1e597ec76575ca10947"}
Sep 16 01:14:49.565095 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.565021    2581 scope.go:117] "RemoveContainer" containerID="d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0"
Sep 16 01:14:49.574555 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.574525    2581 scope.go:117] "RemoveContainer" containerID="d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0"
Sep 16 01:14:49.574879 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:14:49.574850    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0\": container with ID starting with d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0 not found: ID does not exist" containerID="d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0"
Sep 16 01:14:49.574969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.574886    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0"} err="failed to get container status \"d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0\": rpc error: code = NotFound desc = could not find container \"d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0\": container with ID starting with d1db5183e274273be123ae6ba88c95c3691c224e936f7cdc05c571d0aee68bf0 not found: ID does not exist"
Sep 16 01:14:49.598673 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.598638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-model-cache\") pod \"7bd2b445-684a-4473-81a5-5caf97c818e4\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") "
Sep 16 01:14:49.598863 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.598685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd2b445-684a-4473-81a5-5caf97c818e4-tls-certs\") pod \"7bd2b445-684a-4473-81a5-5caf97c818e4\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") "
Sep 16 01:14:49.598863 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.598749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-home\") pod \"7bd2b445-684a-4473-81a5-5caf97c818e4\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") "
Sep 16 01:14:49.598863 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.598772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-tmp-dir\") pod \"7bd2b445-684a-4473-81a5-5caf97c818e4\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") "
Sep 16 01:14:49.598863 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.598800    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkvtz\" (UniqueName: \"kubernetes.io/projected/7bd2b445-684a-4473-81a5-5caf97c818e4-kube-api-access-qkvtz\") pod \"7bd2b445-684a-4473-81a5-5caf97c818e4\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") "
Sep 16 01:14:49.598863 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.598831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-dshm\") pod \"7bd2b445-684a-4473-81a5-5caf97c818e4\" (UID: \"7bd2b445-684a-4473-81a5-5caf97c818e4\") "
Sep 16 01:14:49.599134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.598952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bd2b445-684a-4473-81a5-5caf97c818e4" (UID: "7bd2b445-684a-4473-81a5-5caf97c818e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:14:49.599134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.599011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-home" (OuterVolumeSpecName: "home") pod "7bd2b445-684a-4473-81a5-5caf97c818e4" (UID: "7bd2b445-684a-4473-81a5-5caf97c818e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:14:49.599134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.599038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bd2b445-684a-4473-81a5-5caf97c818e4" (UID: "7bd2b445-684a-4473-81a5-5caf97c818e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:14:49.599295 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.599165    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-home\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:14:49.599295 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.599199    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-tmp-dir\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:14:49.599295 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.599209    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-model-cache\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:14:49.601342 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.601299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-dshm" (OuterVolumeSpecName: "dshm") pod "7bd2b445-684a-4473-81a5-5caf97c818e4" (UID: "7bd2b445-684a-4473-81a5-5caf97c818e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:14:49.601476 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.601371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd2b445-684a-4473-81a5-5caf97c818e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bd2b445-684a-4473-81a5-5caf97c818e4" (UID: "7bd2b445-684a-4473-81a5-5caf97c818e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:14:49.601476 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.601379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd2b445-684a-4473-81a5-5caf97c818e4-kube-api-access-qkvtz" (OuterVolumeSpecName: "kube-api-access-qkvtz") pod "7bd2b445-684a-4473-81a5-5caf97c818e4" (UID: "7bd2b445-684a-4473-81a5-5caf97c818e4"). InnerVolumeSpecName "kube-api-access-qkvtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:14:49.643180 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.643138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:49.701056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.701007    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkvtz\" (UniqueName: \"kubernetes.io/projected/7bd2b445-684a-4473-81a5-5caf97c818e4-kube-api-access-qkvtz\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:14:49.701056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.701044    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd2b445-684a-4473-81a5-5caf97c818e4-dshm\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:14:49.701056 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.701058    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd2b445-684a-4473-81a5-5caf97c818e4-tls-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:14:49.886449 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.886419    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"]
Sep 16 01:14:49.893441 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.893410    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/rate-test-model-0ee1bf-kserve-74d7c8675f-9zccj"]
Sep 16 01:14:49.942158 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:49.942116    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd2b445-684a-4473-81a5-5caf97c818e4" path="/var/lib/kubelet/pods/7bd2b445-684a-4473-81a5-5caf97c818e4/volumes"
Sep 16 01:14:50.410818 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:50.410783    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:51.514104 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:51.514064    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:52.291153 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.291119    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"]
Sep 16 01:14:52.291956 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.291928    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd2b445-684a-4473-81a5-5caf97c818e4" containerName="main"
Sep 16 01:14:52.291956 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.291952    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd2b445-684a-4473-81a5-5caf97c818e4" containerName="main"
Sep 16 01:14:52.292160 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.292034    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd2b445-684a-4473-81a5-5caf97c818e4" containerName="main"
Sep 16 01:14:52.296290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.296269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.298726 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.298704    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ad13e5-serving-cert\""
Sep 16 01:14:52.306699 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.306672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"]
Sep 16 01:14:52.333197 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.333165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:52.428643 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.428611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba2a80e-1426-4bc0-b647-b8a342026bff-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.428859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.428659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-metrics-certs\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.428859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.428724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.428859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.428776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xd4\" (UniqueName: \"kubernetes.io/projected/9ba2a80e-1426-4bc0-b647-b8a342026bff-kube-api-access-98xd4\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.434990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.434952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:52.529428 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.529384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-metrics-certs\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.529952 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.529440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.529952 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.529524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xd4\" (UniqueName: \"kubernetes.io/projected/9ba2a80e-1426-4bc0-b647-b8a342026bff-kube-api-access-98xd4\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.529952 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:14:52.529572    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-serving-cert: secret "maas-api-e2e-api-a-ad13e5-serving-cert" not found
Sep 16 01:14:52.529952 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:14:52.529647    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls podName:9ba2a80e-1426-4bc0-b647-b8a342026bff nodeName:}" failed. No retries permitted until 2026-09-16 01:14:53.029622956 +0000 UTC m=+1911.572759929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls") pod "maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4" (UID: "9ba2a80e-1426-4bc0-b647-b8a342026bff") : secret "maas-api-e2e-api-a-ad13e5-serving-cert" not found
Sep 16 01:14:52.529952 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.529694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba2a80e-1426-4bc0-b647-b8a342026bff-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.530477 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.530452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba2a80e-1426-4bc0-b647-b8a342026bff-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.532093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.532069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-metrics-certs\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.539711 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.539687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xd4\" (UniqueName: \"kubernetes.io/projected/9ba2a80e-1426-4bc0-b647-b8a342026bff-kube-api-access-98xd4\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:52.775305 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.775277    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:52.869286 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:52.869254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:53.035269 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.035170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:53.037588 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.037570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls\") pod \"maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:53.061100 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.061066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:53.161220 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.161179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:53.211217 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.211173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:53.291755 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.291678    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:53.484622 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.484596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"]
Sep 16 01:14:53.486295 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:14:53.486270    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba2a80e_1426_4bc0_b647_b8a342026bff.slice/crio-9f9456539754adbfb2528e1326743443c09a152f3616bca616dc4d51dcf217fb WatchSource:0}: Error finding container 9f9456539754adbfb2528e1326743443c09a152f3616bca616dc4d51dcf217fb: Status 404 returned error can't find the container with id 9f9456539754adbfb2528e1326743443c09a152f3616bca616dc4d51dcf217fb
Sep 16 01:14:53.590058 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:53.589967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4" event={"ID":"9ba2a80e-1426-4bc0-b647-b8a342026bff","Type":"ContainerStarted","Data":"9f9456539754adbfb2528e1326743443c09a152f3616bca616dc4d51dcf217fb"}
Sep 16 01:14:54.005170 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:54.005085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:54.597237 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:54.597201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4" event={"ID":"9ba2a80e-1426-4bc0-b647-b8a342026bff","Type":"ContainerStarted","Data":"c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e"}
Sep 16 01:14:54.597769 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:54.597360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:14:54.616260 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:54.616207    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4" podStartSLOduration=2.330034333 podStartE2EDuration="2.616191366s" totalImagesPullingTime="286.157033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:54.614663165 +0000 UTC m=+1913.157800140" watchObservedRunningTime="2026-09-16 01:14:54.616191366 +0000 UTC m=+1913.159328341"
Sep 16 01:14:54.793830 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:54.793779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:55.685515 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:55.685458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:56.433822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:56.433790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:57.135288 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:57.135256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:59.095768 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:59.095729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:59.407888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:59.407804    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:14:59.560634 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:14:59.560600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:00.157918 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.157877    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"]
Sep 16 01:15:00.162637 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.162611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"
Sep 16 01:15:00.165144 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.165122    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:15:00.168135 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.167931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"]
Sep 16 01:15:00.179344 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.179315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"
Sep 16 01:15:00.450579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.450549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"]
Sep 16 01:15:00.452354 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:15:00.452328    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107c413d_ef03_46fc_9ece_ba4ff2c09f1b.slice/crio-64cda55d09bcf127fb90a1f614ca79f19719c9db8ea69b1c67dc6957c29309ed WatchSource:0}: Error finding container 64cda55d09bcf127fb90a1f614ca79f19719c9db8ea69b1c67dc6957c29309ed: Status 404 returned error can't find the container with id 64cda55d09bcf127fb90a1f614ca79f19719c9db8ea69b1c67dc6957c29309ed
Sep 16 01:15:00.608586 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.608551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:15:00.631640 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.631608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln" event={"ID":"107c413d-ef03-46fc-9ece-ba4ff2c09f1b","Type":"ContainerStarted","Data":"64cda55d09bcf127fb90a1f614ca79f19719c9db8ea69b1c67dc6957c29309ed"}
Sep 16 01:15:00.905284 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:00.905242    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:15:02.011280 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:02.011245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:02.690464 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:02.690423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:02.854233 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:02.854183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:03.018581 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:03.018532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:03.921334 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:03.921207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:04.656935 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:04.656898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln" event={"ID":"107c413d-ef03-46fc-9ece-ba4ff2c09f1b","Type":"ContainerStarted","Data":"a89c7ea99e34aea3bf9bb492c2923c73276eec9466998f56a1a1d19d5c905b5b"}
Sep 16 01:15:04.696675 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:04.696628    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:05.412066 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:05.412024    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:05.677566 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:05.676785    2581 generic.go:356] "Generic (PLEG): container finished" podID="107c413d-ef03-46fc-9ece-ba4ff2c09f1b" containerID="a89c7ea99e34aea3bf9bb492c2923c73276eec9466998f56a1a1d19d5c905b5b" exitCode=0
Sep 16 01:15:05.677566 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:05.676898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln" event={"ID":"107c413d-ef03-46fc-9ece-ba4ff2c09f1b","Type":"ContainerDied","Data":"a89c7ea99e34aea3bf9bb492c2923c73276eec9466998f56a1a1d19d5c905b5b"}
Sep 16 01:15:06.219570 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.219533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:06.846390 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.846362    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"
Sep 16 01:15:06.990099 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.988634    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"]
Sep 16 01:15:06.990099 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.989293    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107c413d-ef03-46fc-9ece-ba4ff2c09f1b" containerName="cleanup"
Sep 16 01:15:06.990099 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.989308    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="107c413d-ef03-46fc-9ece-ba4ff2c09f1b" containerName="cleanup"
Sep 16 01:15:06.990099 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.989418    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="107c413d-ef03-46fc-9ece-ba4ff2c09f1b" containerName="cleanup"
Sep 16 01:15:06.993316 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.992997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:06.995527 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.995504    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-195f51-serving-cert\""
Sep 16 01:15:06.998053 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:06.997964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"]
Sep 16 01:15:07.018078 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.018044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:07.076756 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.076599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:07.091517 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.091295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.091517 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.091360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-metrics-certs\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.091517 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.091410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.091517 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.091451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnl77\" (UniqueName: \"kubernetes.io/projected/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-kube-api-access-wnl77\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.180220 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.180141    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:07.192151 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.192118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.192345 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.192168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-metrics-certs\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.192424 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.192369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.192499 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.192438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnl77\" (UniqueName: \"kubernetes.io/projected/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-kube-api-access-wnl77\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.193122 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.193097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.194788 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.194768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.194871 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.194785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-metrics-certs\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.199541 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.199519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnl77\" (UniqueName: \"kubernetes.io/projected/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-kube-api-access-wnl77\") pod \"maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.309774 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.309740    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:07.480506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.476944    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:07.582262 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.582044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:07.604678 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.604646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"]
Sep 16 01:15:07.606734 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:15:07.606701    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbc7248_c53e_4f76_a6a9_3e7b13fb4df5.slice/crio-d4e35c2696e803041bae2ac4218a3db16a21ea5f5154dace4ecb2674c23829fc WatchSource:0}: Error finding container d4e35c2696e803041bae2ac4218a3db16a21ea5f5154dace4ecb2674c23829fc: Status 404 returned error can't find the container with id d4e35c2696e803041bae2ac4218a3db16a21ea5f5154dace4ecb2674c23829fc
Sep 16 01:15:07.694344 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.694286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv" event={"ID":"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5","Type":"ContainerStarted","Data":"d4e35c2696e803041bae2ac4218a3db16a21ea5f5154dace4ecb2674c23829fc"}
Sep 16 01:15:07.695545 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.695517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln" event={"ID":"107c413d-ef03-46fc-9ece-ba4ff2c09f1b","Type":"ContainerDied","Data":"64cda55d09bcf127fb90a1f614ca79f19719c9db8ea69b1c67dc6957c29309ed"}
Sep 16 01:15:07.695545 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.695542    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"
Sep 16 01:15:07.695706 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.695566    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64cda55d09bcf127fb90a1f614ca79f19719c9db8ea69b1c67dc6957c29309ed"
Sep 16 01:15:07.885150 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:07.885036    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:08.391843 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:08.391740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:08.607917 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:08.607885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:08.703822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:08.703728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv" event={"ID":"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5","Type":"ContainerStarted","Data":"209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9"}
Sep 16 01:15:08.703822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:08.703790    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:08.720206 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:08.720150    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv" podStartSLOduration=2.225671631 podStartE2EDuration="2.720130168s" totalImagesPullingTime="494.458537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:08.718603852 +0000 UTC m=+1927.261740826" watchObservedRunningTime="2026-09-16 01:15:08.720130168 +0000 UTC m=+1927.263267142"
Sep 16 01:15:08.837769 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:08.837735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:09.143674 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:09.143634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:09.436804 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:09.436717    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:09.539474 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:09.539441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:10.292826 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:10.292783    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:11.003221 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:11.003179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:12.023575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.023544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:12.684805 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.684769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:12.729072 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.729039    2581 generic.go:356] "Generic (PLEG): container finished" podID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerID="8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a" exitCode=0
Sep 16 01:15:12.729245 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.729115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" event={"ID":"d077d476-a1e1-4476-99b3-f8bc4e431eea","Type":"ContainerDied","Data":"8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a"}
Sep 16 01:15:12.729636 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.729619    2581 scope.go:117] "RemoveContainer" containerID="8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a"
Sep 16 01:15:12.730784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.730756    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:15:12.730854 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.730823    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:15:12.730894 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.730854    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:15:12.775017 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.774983    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:12.877624 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:12.877589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:13.729624 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:13.729591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:13.736084 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:13.736041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" event={"ID":"d077d476-a1e1-4476-99b3-f8bc4e431eea","Type":"ContainerStarted","Data":"a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6"}
Sep 16 01:15:13.896512 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:13.896455    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:15:13.897874 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:13.897844    2581 patch_prober.go:28] interesting pod/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:15:13.898023 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:13.897904    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 01:15:14.538013 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:14.537977    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:14.716390 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:14.716354    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:15:14.897370 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:14.897263    2581 patch_prober.go:28] interesting pod/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:15:14.897370 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:14.897328    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 01:15:15.318574 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:15.318538    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:15.897647 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:15.897603    2581 patch_prober.go:28] interesting pod/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:15:15.898054 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:15.897662    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 01:15:16.075297 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:16.075260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:16.790195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:16.790161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:16.897686 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:16.897646    2581 patch_prober.go:28] interesting pod/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:15:16.898106 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:16.897718    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 01:15:17.429643 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.429607    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"]
Sep 16 01:15:17.437761 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.437731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.440696 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.440355    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-00f5c9-serving-cert\""
Sep 16 01:15:17.442358 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.442134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"]
Sep 16 01:15:17.479507 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.468977    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:17.557599 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.557557    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:17.605126 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.605091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f308c3d9-db73-4912-8998-758f2ec6325c-openshift-service-ca\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.605338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.605143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8msvv\" (UniqueName: \"kubernetes.io/projected/f308c3d9-db73-4912-8998-758f2ec6325c-kube-api-access-8msvv\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.605338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.605189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.605338 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.605271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-metrics-certs\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.706195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.706082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f308c3d9-db73-4912-8998-758f2ec6325c-openshift-service-ca\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.706195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.706133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8msvv\" (UniqueName: \"kubernetes.io/projected/f308c3d9-db73-4912-8998-758f2ec6325c-kube-api-access-8msvv\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.706195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.706170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.706195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.706203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-metrics-certs\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.706595 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:15:17.706342    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-serving-cert: secret "maas-api-e2e-api-b-00f5c9-serving-cert" not found
Sep 16 01:15:17.706595 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:15:17.706432    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls podName:f308c3d9-db73-4912-8998-758f2ec6325c nodeName:}" failed. No retries permitted until 2026-09-16 01:15:18.206411059 +0000 UTC m=+1936.749548019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls") pod "maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc" (UID: "f308c3d9-db73-4912-8998-758f2ec6325c") : secret "maas-api-e2e-api-b-00f5c9-serving-cert" not found
Sep 16 01:15:17.706908 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.706884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f308c3d9-db73-4912-8998-758f2ec6325c-openshift-service-ca\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.708750 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.708723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-metrics-certs\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.714130 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.714096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8msvv\" (UniqueName: \"kubernetes.io/projected/f308c3d9-db73-4912-8998-758f2ec6325c-kube-api-access-8msvv\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:17.811293 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.811257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:17.896724 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.896691    2581 patch_prober.go:28] interesting pod/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:15:17.896897 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.896747    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 01:15:17.915054 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:17.915024    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:18.019547 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.019516    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:18.211566 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.211474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:18.214044 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.214025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls\") pod \"maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:18.308036 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.307947    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:18.353200 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.353166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:18.544209 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.544172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:15:18.645989 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.645964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"]
Sep 16 01:15:18.649503 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:15:18.649448    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf308c3d9_db73_4912_8998_758f2ec6325c.slice/crio-d6c559a0c786fa0e72fea691dd335dc83ae71c0b88b2e1d9c20070b6c927eefd WatchSource:0}: Error finding container d6c559a0c786fa0e72fea691dd335dc83ae71c0b88b2e1d9c20070b6c927eefd: Status 404 returned error can't find the container with id d6c559a0c786fa0e72fea691dd335dc83ae71c0b88b2e1d9c20070b6c927eefd
Sep 16 01:15:18.770251 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.770118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc" event={"ID":"f308c3d9-db73-4912-8998-758f2ec6325c","Type":"ContainerStarted","Data":"d6c559a0c786fa0e72fea691dd335dc83ae71c0b88b2e1d9c20070b6c927eefd"}
Sep 16 01:15:18.897392 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.897295    2581 patch_prober.go:28] interesting pod/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:15:18.897392 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:18.897360    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 01:15:19.120867 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:19.120833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:19.778270 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:19.778232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc" event={"ID":"f308c3d9-db73-4912-8998-758f2ec6325c","Type":"ContainerStarted","Data":"1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15"}
Sep 16 01:15:19.778567 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:19.778307    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:19.794629 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:19.794570    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc" podStartSLOduration=2.370780798 podStartE2EDuration="2.794552063s" totalImagesPullingTime="423.771265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:19.79356844 +0000 UTC m=+1938.336705415" watchObservedRunningTime="2026-09-16 01:15:19.794552063 +0000 UTC m=+1938.337689035"
Sep 16 01:15:19.902332 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:19.902300    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:15:19.903397 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:19.903369    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:15:23.940756 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:23.940722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:24.041538 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:24.041507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:25.790400 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:25.790370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:15:26.384980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:26.384945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:15:27.855946 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:27.855908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:29.200617 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:29.200577    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:30.101179 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:30.101136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:31.006613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:31.006575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:31.759457 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:31.759421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:32.377036 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:32.377006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:32.581634 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:32.581604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:32.860612 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:32.860578    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:33.156510 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:33.156382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:33.810743 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:33.810710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:34.633204 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:34.633170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:35.297049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:35.297000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:36.134742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:36.134710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:36.914953 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:36.914916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:37.594942 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:37.594907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:38.242741 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:38.242700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:39.025739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:39.025687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:39.685598 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:39.685558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:40.475423 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:40.475386    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:41.252011 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:41.251960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:42.096744 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:42.096709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:45.923673 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:45.923623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:50.352150 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:50.352110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:51.524735 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.524696    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"]
Sep 16 01:15:51.525195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.524960    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4" podUID="9ba2a80e-1426-4bc0-b647-b8a342026bff" containerName="maas-api" containerID="cri-o://c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e" gracePeriod=30
Sep 16 01:15:51.552645 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.552609    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:51.621718 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.621659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:51.720041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.719996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:51.772282 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.772248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:51.779546 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.779467    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:15:51.962867 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.962825    2581 generic.go:356] "Generic (PLEG): container finished" podID="9ba2a80e-1426-4bc0-b647-b8a342026bff" containerID="c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e" exitCode=0
Sep 16 01:15:51.962867 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.962869    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"
Sep 16 01:15:51.963092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.962900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4" event={"ID":"9ba2a80e-1426-4bc0-b647-b8a342026bff","Type":"ContainerDied","Data":"c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e"}
Sep 16 01:15:51.963092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.962951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4" event={"ID":"9ba2a80e-1426-4bc0-b647-b8a342026bff","Type":"ContainerDied","Data":"9f9456539754adbfb2528e1326743443c09a152f3616bca616dc4d51dcf217fb"}
Sep 16 01:15:51.963092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.962976    2581 scope.go:117] "RemoveContainer" containerID="c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e"
Sep 16 01:15:51.967489 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.967448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls\") pod \"9ba2a80e-1426-4bc0-b647-b8a342026bff\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") "
Sep 16 01:15:51.967626 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.967524    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba2a80e-1426-4bc0-b647-b8a342026bff-openshift-service-ca\") pod \"9ba2a80e-1426-4bc0-b647-b8a342026bff\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") "
Sep 16 01:15:51.967626 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.967613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98xd4\" (UniqueName: \"kubernetes.io/projected/9ba2a80e-1426-4bc0-b647-b8a342026bff-kube-api-access-98xd4\") pod \"9ba2a80e-1426-4bc0-b647-b8a342026bff\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") "
Sep 16 01:15:51.967730 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.967659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-metrics-certs\") pod \"9ba2a80e-1426-4bc0-b647-b8a342026bff\" (UID: \"9ba2a80e-1426-4bc0-b647-b8a342026bff\") "
Sep 16 01:15:51.968356 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.967924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba2a80e-1426-4bc0-b647-b8a342026bff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ba2a80e-1426-4bc0-b647-b8a342026bff" (UID: "9ba2a80e-1426-4bc0-b647-b8a342026bff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:15:51.968356 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.968083    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba2a80e-1426-4bc0-b647-b8a342026bff-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:15:51.969832 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.969794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ba2a80e-1426-4bc0-b647-b8a342026bff" (UID: "9ba2a80e-1426-4bc0-b647-b8a342026bff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:15:51.970430 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.970408    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba2a80e-1426-4bc0-b647-b8a342026bff-kube-api-access-98xd4" (OuterVolumeSpecName: "kube-api-access-98xd4") pod "9ba2a80e-1426-4bc0-b647-b8a342026bff" (UID: "9ba2a80e-1426-4bc0-b647-b8a342026bff"). InnerVolumeSpecName "kube-api-access-98xd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:15:51.970565 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.970417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ba2a80e-1426-4bc0-b647-b8a342026bff" (UID: "9ba2a80e-1426-4bc0-b647-b8a342026bff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:15:51.974369 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.974345    2581 scope.go:117] "RemoveContainer" containerID="c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e"
Sep 16 01:15:51.974786 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:15:51.974762    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e\": container with ID starting with c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e not found: ID does not exist" containerID="c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e"
Sep 16 01:15:51.974860 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:51.974797    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e"} err="failed to get container status \"c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e\": rpc error: code = NotFound desc = could not find container \"c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e\": container with ID starting with c864f92f227f8da2f875207bb06eaf15561f9c1cda0d682e41850a7452a57d7e not found: ID does not exist"
Sep 16 01:15:52.069608 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:52.069492    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xd4\" (UniqueName: \"kubernetes.io/projected/9ba2a80e-1426-4bc0-b647-b8a342026bff-kube-api-access-98xd4\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:15:52.069608 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:52.069537    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:15:52.069608 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:52.069555    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba2a80e-1426-4bc0-b647-b8a342026bff-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:15:52.138466 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:52.138421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:52.288232 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:52.288198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"]
Sep 16 01:15:52.291495 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:52.291448    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad13e5-66787b8ccf-ndrk4"]
Sep 16 01:15:52.311586 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:52.311558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:53.227976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:53.227934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:53.888292 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:53.888254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:53.942679 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:53.942602    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba2a80e-1426-4bc0-b647-b8a342026bff" path="/var/lib/kubelet/pods/9ba2a80e-1426-4bc0-b647-b8a342026bff/volumes"
Sep 16 01:15:54.522359 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:54.522303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:54.688213 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:54.688161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:55.317324 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:55.317289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:55.993438 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:55.993399    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:56.866646 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:56.866612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:58.025836 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:58.025795    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:58.125946 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:58.125898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:58.270553 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:58.270518    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"]
Sep 16 01:15:58.270812 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:58.270773    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy" containerID="cri-o://a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6" gracePeriod=30
Sep 16 01:15:58.292941 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:58.292851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:58.728104 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:58.728001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:15:59.287542 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:59.287502    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:15:59.905933 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:15:59.905901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:00.595473 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:00.595443    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:01.327129 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:01.327090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:01.995050 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:01.995009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:02.661645 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:02.661608    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:03.424790 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.424750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:03.524727 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.524700    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:16:03.564595 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-envoy\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.564595 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564543    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d077d476-a1e1-4476-99b3-f8bc4e431eea-istiod-ca-cert\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.564813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-socket\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.564813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-token\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.564813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-podinfo\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.564813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564723    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-certs\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.564813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-data\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.564813 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-credential-socket\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.565106 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.564830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtk6p\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-kube-api-access-vtk6p\") pod \"d077d476-a1e1-4476-99b3-f8bc4e431eea\" (UID: \"d077d476-a1e1-4476-99b3-f8bc4e431eea\") "
Sep 16 01:16:03.565257 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.565180    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:03.565257 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.565046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d077d476-a1e1-4476-99b3-f8bc4e431eea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:03.565257 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.565218    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:03.565500 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.565420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-data" (OuterVolumeSpecName: "istio-data") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:03.565500 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.565436    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:03.566944 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.566907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:03.567174 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.567157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-token" (OuterVolumeSpecName: "istio-token") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:03.567248 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.567234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-kube-api-access-vtk6p" (OuterVolumeSpecName: "kube-api-access-vtk6p") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "kube-api-access-vtk6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:03.567362 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.567346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d077d476-a1e1-4476-99b3-f8bc4e431eea" (UID: "d077d476-a1e1-4476-99b3-f8bc4e431eea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:16:03.666596 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666559    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666596 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666590    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d077d476-a1e1-4476-99b3-f8bc4e431eea-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666596 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666601    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666834 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666609    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666834 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666617    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666834 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666625    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666834 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666634    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666834 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666641    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d077d476-a1e1-4476-99b3-f8bc4e431eea-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:03.666834 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:03.666649    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtk6p\" (UniqueName: \"kubernetes.io/projected/d077d476-a1e1-4476-99b3-f8bc4e431eea-kube-api-access-vtk6p\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:04.040470 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.040431    2581 generic.go:356] "Generic (PLEG): container finished" podID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerID="a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6" exitCode=0
Sep 16 01:16:04.040668 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.040513    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"
Sep 16 01:16:04.040668 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.040519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" event={"ID":"d077d476-a1e1-4476-99b3-f8bc4e431eea","Type":"ContainerDied","Data":"a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6"}
Sep 16 01:16:04.040668 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.040581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl" event={"ID":"d077d476-a1e1-4476-99b3-f8bc4e431eea","Type":"ContainerDied","Data":"65c358b97fce7f0f6a11aa003c8e7a7b9588cf4985d7d63e2f4e3792b5c7c38e"}
Sep 16 01:16:04.040668 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.040604    2581 scope.go:117] "RemoveContainer" containerID="a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6"
Sep 16 01:16:04.049929 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.049914    2581 scope.go:117] "RemoveContainer" containerID="8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a"
Sep 16 01:16:04.058082 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.058054    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"]
Sep 16 01:16:04.058784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.058758    2581 scope.go:117] "RemoveContainer" containerID="a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6"
Sep 16 01:16:04.059110 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:16:04.059077    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6\": container with ID starting with a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6 not found: ID does not exist" containerID="a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6"
Sep 16 01:16:04.059110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.059109    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6"} err="failed to get container status \"a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6\": rpc error: code = NotFound desc = could not find container \"a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6\": container with ID starting with a27dabffd2425cf0840cb2217d1e508882bbdab81b9b5911e947bbcc244a63f6 not found: ID does not exist"
Sep 16 01:16:04.059110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.059128    2581 scope.go:117] "RemoveContainer" containerID="8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a"
Sep 16 01:16:04.059981 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:16:04.059946    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a\": container with ID starting with 8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a not found: ID does not exist" containerID="8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a"
Sep 16 01:16:04.060093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.059988    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a"} err="failed to get container status \"8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a\": rpc error: code = NotFound desc = could not find container \"8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a\": container with ID starting with 8b22a89e6a7a8bcf105a26fd93c6fdabe442bcaba9cbc1edd9ecda5315fc9f5a not found: ID does not exist"
Sep 16 01:16:04.062077 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.062055    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ad13e5-openshift-default-5466b9c866-8klnl"]
Sep 16 01:16:04.130736 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:04.130687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:05.941632 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:05.941596    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" path="/var/lib/kubelet/pods/d077d476-a1e1-4476-99b3-f8bc4e431eea/volumes"
Sep 16 01:16:08.527699 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:08.527662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:14.078340 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.078303    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"]
Sep 16 01:16:14.078795 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.078578    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc" podUID="f308c3d9-db73-4912-8998-758f2ec6325c" containerName="maas-api" containerID="cri-o://1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15" gracePeriod=30
Sep 16 01:16:14.104406 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.104372    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:14.204865 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.204818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:14.295310 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.295274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:14.351348 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.351315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:14.354730 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.354706    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:16:14.474086 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.474050    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls\") pod \"f308c3d9-db73-4912-8998-758f2ec6325c\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") "
Sep 16 01:16:14.474290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.474106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-metrics-certs\") pod \"f308c3d9-db73-4912-8998-758f2ec6325c\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") "
Sep 16 01:16:14.474290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.474151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8msvv\" (UniqueName: \"kubernetes.io/projected/f308c3d9-db73-4912-8998-758f2ec6325c-kube-api-access-8msvv\") pod \"f308c3d9-db73-4912-8998-758f2ec6325c\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") "
Sep 16 01:16:14.474290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.474216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f308c3d9-db73-4912-8998-758f2ec6325c-openshift-service-ca\") pod \"f308c3d9-db73-4912-8998-758f2ec6325c\" (UID: \"f308c3d9-db73-4912-8998-758f2ec6325c\") "
Sep 16 01:16:14.474590 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.474527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f308c3d9-db73-4912-8998-758f2ec6325c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f308c3d9-db73-4912-8998-758f2ec6325c" (UID: "f308c3d9-db73-4912-8998-758f2ec6325c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:14.474657 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.474646    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f308c3d9-db73-4912-8998-758f2ec6325c-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:14.476399 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.476372    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f308c3d9-db73-4912-8998-758f2ec6325c" (UID: "f308c3d9-db73-4912-8998-758f2ec6325c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:16:14.476557 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.476415    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f308c3d9-db73-4912-8998-758f2ec6325c-kube-api-access-8msvv" (OuterVolumeSpecName: "kube-api-access-8msvv") pod "f308c3d9-db73-4912-8998-758f2ec6325c" (UID: "f308c3d9-db73-4912-8998-758f2ec6325c"). InnerVolumeSpecName "kube-api-access-8msvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:14.476557 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.476459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f308c3d9-db73-4912-8998-758f2ec6325c" (UID: "f308c3d9-db73-4912-8998-758f2ec6325c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:16:14.576154 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.576104    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8msvv\" (UniqueName: \"kubernetes.io/projected/f308c3d9-db73-4912-8998-758f2ec6325c-kube-api-access-8msvv\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:14.576154 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.576150    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:14.576154 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.576161    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f308c3d9-db73-4912-8998-758f2ec6325c-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:16:14.675233 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.675142    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:14.779198 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:14.779164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:15.111381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.111341    2581 generic.go:356] "Generic (PLEG): container finished" podID="f308c3d9-db73-4912-8998-758f2ec6325c" containerID="1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15" exitCode=0
Sep 16 01:16:15.111837 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.111404    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"
Sep 16 01:16:15.111837 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.111429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc" event={"ID":"f308c3d9-db73-4912-8998-758f2ec6325c","Type":"ContainerDied","Data":"1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15"}
Sep 16 01:16:15.111837 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.111502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc" event={"ID":"f308c3d9-db73-4912-8998-758f2ec6325c","Type":"ContainerDied","Data":"d6c559a0c786fa0e72fea691dd335dc83ae71c0b88b2e1d9c20070b6c927eefd"}
Sep 16 01:16:15.111837 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.111528    2581 scope.go:117] "RemoveContainer" containerID="1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15"
Sep 16 01:16:15.121397 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.121379    2581 scope.go:117] "RemoveContainer" containerID="1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15"
Sep 16 01:16:15.121719 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:16:15.121700    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15\": container with ID starting with 1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15 not found: ID does not exist" containerID="1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15"
Sep 16 01:16:15.121793 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.121728    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15"} err="failed to get container status \"1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15\": rpc error: code = NotFound desc = could not find container \"1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15\": container with ID starting with 1e9972c44bedf121689a6fde1206b7d4cc1ee818d5ced34deff57f06f0a14e15 not found: ID does not exist"
Sep 16 01:16:15.132775 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.132749    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"]
Sep 16 01:16:15.136356 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.136331    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00f5c9-559f97cbb8-9npcc"]
Sep 16 01:16:15.513250 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.513215    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:15.942242 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:15.942153    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f308c3d9-db73-4912-8998-758f2ec6325c" path="/var/lib/kubelet/pods/f308c3d9-db73-4912-8998-758f2ec6325c/volumes"
Sep 16 01:16:16.310851 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:16.310776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:16.963819 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:16.963772    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:17.351575 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:17.351534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:17.404747 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:17.404705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:17.599652 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:17.599615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:18.430331 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:18.430286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:20.243116 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:20.243081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:20.343394 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:20.343357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:20.504498 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:20.504398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:21.898699 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:21.898664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:22.554581 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:22.554540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:23.318253 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:23.318218    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:23.974125 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:23.974094    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:24.644068 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:24.644029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:24.877859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:24.877825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:16:25.411630 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:25.411596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:29.443191 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:29.443153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:30.974838 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:30.974800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:31.548290 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.548250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"]
Sep 16 01:16:31.550146 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.550122    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f39e09-29825355-z95ln"]
Sep 16 01:16:31.551526 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.551499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:31.613342 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.613305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:31.667272 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.667240    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:31.734852 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.734817    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:31.782784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.782746    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:31.942175 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:31.942100    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="107c413d-ef03-46fc-9ece-ba4ff2c09f1b" path="/var/lib/kubelet/pods/107c413d-ef03-46fc-9ece-ba4ff2c09f1b/volumes"
Sep 16 01:16:32.103198 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:32.103166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:32.321892 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:32.321859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:32.978943 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:32.978896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:33.079908 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.079866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:33.340224 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340190    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"]
Sep 16 01:16:33.340835 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340818    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba2a80e-1426-4bc0-b647-b8a342026bff" containerName="maas-api"
Sep 16 01:16:33.340885 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340838    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba2a80e-1426-4bc0-b647-b8a342026bff" containerName="maas-api"
Sep 16 01:16:33.340885 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340865    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy"
Sep 16 01:16:33.340885 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340874    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy"
Sep 16 01:16:33.340980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340894    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f308c3d9-db73-4912-8998-758f2ec6325c" containerName="maas-api"
Sep 16 01:16:33.340980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340903    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f308c3d9-db73-4912-8998-758f2ec6325c" containerName="maas-api"
Sep 16 01:16:33.340980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy"
Sep 16 01:16:33.340980 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.340941    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy"
Sep 16 01:16:33.341108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.341024    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy"
Sep 16 01:16:33.341108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.341035    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba2a80e-1426-4bc0-b647-b8a342026bff" containerName="maas-api"
Sep 16 01:16:33.341108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.341044    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d077d476-a1e1-4476-99b3-f8bc4e431eea" containerName="istio-proxy"
Sep 16 01:16:33.341108 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.341056    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f308c3d9-db73-4912-8998-758f2ec6325c" containerName="maas-api"
Sep 16 01:16:33.346365 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.346340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.349104 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.349080    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f46871-openshift-default-dockercfg-l55tc\""
Sep 16 01:16:33.355808 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.355782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"]
Sep 16 01:16:33.369345 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.369294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:33.462247 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-credential-socket\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462442 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-certs\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462442 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5wc4\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-kube-api-access-r5wc4\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462442 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-data\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462589 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-socket\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462589 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-envoy\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462698 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b608f9-631a-4902-a9e6-81a57f218373-istio-podinfo\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462698 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-istio-token\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.462794 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.462694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b608f9-631a-4902-a9e6-81a57f218373-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.563541 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-socket\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.563541 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-envoy\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.563822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b608f9-631a-4902-a9e6-81a57f218373-istio-podinfo\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.563822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-istio-token\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.563822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b608f9-631a-4902-a9e6-81a57f218373-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.563822 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-credential-socket\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564034 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-certs\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564034 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-socket\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564034 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wc4\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-kube-api-access-r5wc4\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564034 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.563997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-data\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564264 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.564147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-credential-socket\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564321 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.564272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-certs\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564376 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.564356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b608f9-631a-4902-a9e6-81a57f218373-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.564427 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.564394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-data\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.566073 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.566044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-envoy\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.566519 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.566501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b608f9-631a-4902-a9e6-81a57f218373-istio-podinfo\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.573312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.573286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-istio-token\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.573400 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.573371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wc4\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-kube-api-access-r5wc4\") pod \"e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") " pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.659914 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.659826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:33.675990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.675961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:33.778788 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.778748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:33.950954 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.950925    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"]
Sep 16 01:16:33.952397 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:16:33.952366    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b608f9_631a_4902_a9e6_81a57f218373.slice/crio-d1210f809570dffce1f3e342a94551a61621948ef8d4eba830b467bf4c4191ad WatchSource:0}: Error finding container d1210f809570dffce1f3e342a94551a61621948ef8d4eba830b467bf4c4191ad: Status 404 returned error can't find the container with id d1210f809570dffce1f3e342a94551a61621948ef8d4eba830b467bf4c4191ad
Sep 16 01:16:33.954952 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.954915    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:16:33.955049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.954989    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:16:33.955049 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:33.955021    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:16:34.218807 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:34.218693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" event={"ID":"46b608f9-631a-4902-a9e6-81a57f218373","Type":"ContainerStarted","Data":"f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378"}
Sep 16 01:16:34.218807 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:34.218758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" event={"ID":"46b608f9-631a-4902-a9e6-81a57f218373","Type":"ContainerStarted","Data":"d1210f809570dffce1f3e342a94551a61621948ef8d4eba830b467bf4c4191ad"}
Sep 16 01:16:34.243411 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:34.243360    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podStartSLOduration=1.243338724 podStartE2EDuration="1.243338724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:34.242514941 +0000 UTC m=+2012.785651917" watchObservedRunningTime="2026-09-16 01:16:34.243338724 +0000 UTC m=+2012.786475699"
Sep 16 01:16:34.508825 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:34.508729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:34.660327 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:34.660279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:35.284805 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:35.284752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:35.665419 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:35.665343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:36.232038 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:36.231966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:36.232923 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:36.232903    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:16:36.235050 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:36.235028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:36.333969 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:36.333924    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:37.179104 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:37.179070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:37.899564 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:37.899533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:38.039443 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:38.039400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:38.138936 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:38.138901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:38.281910 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:38.281881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:38.799599 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:38.799562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:39.711921 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:39.711888    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:40.527727 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:40.527683    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:40.628508 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:40.628456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:40.678939 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:40.678907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:40.991682 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:40.991607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:41.085319 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:41.085286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:41.386961 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:41.386923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:42.383279 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:42.383242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:43.042432 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:43.042399    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:43.823591 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:43.823555    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:44.553878 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:44.553840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:45.288085 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:45.288054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:46.679449 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:46.679413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:16:48.253551 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:48.253507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:48.815679 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:48.815635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:48.880738 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:48.880692    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:48.981218 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:48.981175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:49.044475 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:49.044433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:49.374138 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:49.374092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:49.474324 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:49.474293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:50.215117 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:50.215074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:50.893015 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:50.892969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:51.547781 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:51.547743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:52.417700 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:52.417658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:53.308514 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:53.308444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:53.959400 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:53.959364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:54.710850 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:54.710810    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:55.100063 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:55.100029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:55.200542 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:55.200505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:55.352281 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:55.352187    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"]
Sep 16 01:16:55.352457 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:55.352419    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy" containerID="cri-o://5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a" gracePeriod=30
Sep 16 01:16:55.374093 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:55.374059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:55.470745 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:55.470710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:56.103727 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:56.103687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:56.201598 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:56.201567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:57.103447 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:57.103411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:57.854616 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:57.854567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:58.613208 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:58.613175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:59.237145 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:59.237113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:16:59.365784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:59.365746    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b608f9-631a-4902-a9e6-81a57f218373" containerID="f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378" exitCode=0
Sep 16 01:16:59.365784 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:59.365786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" event={"ID":"46b608f9-631a-4902-a9e6-81a57f218373","Type":"ContainerDied","Data":"f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378"}
Sep 16 01:16:59.366292 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:59.366271    2581 scope.go:117] "RemoveContainer" containerID="f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378"
Sep 16 01:16:59.367477 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:59.367435    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:16:59.367587 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:59.367531    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:16:59.367587 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:16:59.367563    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 01:17:00.116512 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.116443    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:00.377047 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.375821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" event={"ID":"46b608f9-631a-4902-a9e6-81a57f218373","Type":"ContainerStarted","Data":"c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab"}
Sep 16 01:17:00.627848 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.627779    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:17:00.683509 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.677940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:17:00.683509 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.678078    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 01:17:00.683509 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.678134    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 01:17:00.725122 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f419f-4ff6-4511-801f-e50af17337b3-istiod-ca-cert\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725122 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-envoy\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f419f-4ff6-4511-801f-e50af17337b3-istio-podinfo\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-certs\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-data\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725280    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-credential-socket\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725308    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-istio-token\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725335    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-socket\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725381 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725381    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6mt8\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-kube-api-access-d6mt8\") pod \"484f419f-4ff6-4511-801f-e50af17337b3\" (UID: \"484f419f-4ff6-4511-801f-e50af17337b3\") "
Sep 16 01:17:00.725783 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:00.725783 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725614    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:00.725783 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484f419f-4ff6-4511-801f-e50af17337b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:17:00.725783 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:00.725965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725788    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484f419f-4ff6-4511-801f-e50af17337b3-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.725965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725808    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.725965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725822    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.725965 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.725943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:00.727653 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.727629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:00.727790 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.727771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-kube-api-access-d6mt8" (OuterVolumeSpecName: "kube-api-access-d6mt8") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "kube-api-access-d6mt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:17:00.727872 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.727859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/484f419f-4ff6-4511-801f-e50af17337b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:17:00.727972 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.727954    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "484f419f-4ff6-4511-801f-e50af17337b3" (UID: "484f419f-4ff6-4511-801f-e50af17337b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:17:00.795046 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.795010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:00.827304 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.827269    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.827304 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.827299    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.827304 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.827309    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.827592 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.827318    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6mt8\" (UniqueName: \"kubernetes.io/projected/484f419f-4ff6-4511-801f-e50af17337b3-kube-api-access-d6mt8\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.827592 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.827328    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484f419f-4ff6-4511-801f-e50af17337b3-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.827592 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:00.827337    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484f419f-4ff6-4511-801f-e50af17337b3-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:01.386232 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.386195    2581 generic.go:356] "Generic (PLEG): container finished" podID="484f419f-4ff6-4511-801f-e50af17337b3" containerID="5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a" exitCode=0
Sep 16 01:17:01.386798 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.386252    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"
Sep 16 01:17:01.386798 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.386294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" event={"ID":"484f419f-4ff6-4511-801f-e50af17337b3","Type":"ContainerDied","Data":"5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a"}
Sep 16 01:17:01.386798 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.386358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp" event={"ID":"484f419f-4ff6-4511-801f-e50af17337b3","Type":"ContainerDied","Data":"6f95efa9bf9374a6640a48836bf10a3218dd6307cd38d95f7cf524c5a7ded770"}
Sep 16 01:17:01.386798 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.386392    2581 scope.go:117] "RemoveContainer" containerID="5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a"
Sep 16 01:17:01.396398 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.396380    2581 scope.go:117] "RemoveContainer" containerID="532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e"
Sep 16 01:17:01.405535 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.405507    2581 scope.go:117] "RemoveContainer" containerID="5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a"
Sep 16 01:17:01.405923 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:17:01.405901    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a\": container with ID starting with 5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a not found: ID does not exist" containerID="5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a"
Sep 16 01:17:01.406015 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.405936    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a"} err="failed to get container status \"5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a\": rpc error: code = NotFound desc = could not find container \"5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a\": container with ID starting with 5f28ddec326b355008e1e9b794583c507415f763c11809ace1b0317e0bed5e3a not found: ID does not exist"
Sep 16 01:17:01.406015 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.405962    2581 scope.go:117] "RemoveContainer" containerID="532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e"
Sep 16 01:17:01.406301 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:17:01.406273    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e\": container with ID starting with 532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e not found: ID does not exist" containerID="532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e"
Sep 16 01:17:01.406372 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.406306    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e"} err="failed to get container status \"532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e\": rpc error: code = NotFound desc = could not find container \"532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e\": container with ID starting with 532802231bfee74186bb4a22ae65e7973fe0a82486d9e6d0731dd421e58d7e7e not found: ID does not exist"
Sep 16 01:17:01.407792 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.407768    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"]
Sep 16 01:17:01.409794 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.409775    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-45446b-openshift-default-64c94999cb-fpwlp"]
Sep 16 01:17:01.660898 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.660791    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 01:17:01.660898 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.660880    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 01:17:01.942439 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:01.942359    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484f419f-4ff6-4511-801f-e50af17337b3" path="/var/lib/kubelet/pods/484f419f-4ff6-4511-801f-e50af17337b3/volumes"
Sep 16 01:17:02.573733 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:02.573691    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:17:02.660668 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:02.660629    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 01:17:02.660872 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:02.660719    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 01:17:03.660597 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:03.660555    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 01:17:03.661110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:03.660634    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 01:17:04.660860 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:04.660810    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 01:17:04.661388 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:04.660888    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 01:17:05.661650 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:05.661123    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 01:17:05.661650 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:05.661237    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 01:17:06.232785 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:06.232747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:17:06.664996 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:06.664962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:17:06.665910 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:06.665890    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:17:07.356840 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:07.356805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:07.418526 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:07.418453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:07.518205 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:07.518171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:07.574594 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:07.574559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:07.957348 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:07.957309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:08.113100 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:08.113072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:08.898134 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:08.898100    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:09.606495 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:09.606451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:10.256915 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:10.256877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:11.486069 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:11.486036    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:17:13.581659 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:13.581626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:13.641175 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:13.641126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:13.838294 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:13.838198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"]
Sep 16 01:17:13.838517 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:13.838467    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy" containerID="cri-o://c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab" gracePeriod=30
Sep 16 01:17:13.874504 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:13.874447    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:18.027085 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:18.027048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:18.913155 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:18.913120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:19.101138 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.101066    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:17:19.203341 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b608f9-631a-4902-a9e6-81a57f218373-istio-podinfo\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-credential-socket\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-certs\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-data\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-socket\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203577 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-istio-token\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b608f9-631a-4902-a9e6-81a57f218373-istiod-ca-cert\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5wc4\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-kube-api-access-r5wc4\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-envoy\") pod \"46b608f9-631a-4902-a9e6-81a57f218373\" (UID: \"46b608f9-631a-4902-a9e6-81a57f218373\") "
Sep 16 01:17:19.203859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203772    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:19.203859 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-data" (OuterVolumeSpecName: "istio-data") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:19.204126 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.203855    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:19.204126 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.204059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b608f9-631a-4902-a9e6-81a57f218373-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:17:19.204126 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.204064    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:19.204301 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.204163    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-credential-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.204301 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.204183    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.204301 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.204195    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-data\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.206132 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.206103    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-kube-api-access-r5wc4" (OuterVolumeSpecName: "kube-api-access-r5wc4") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "kube-api-access-r5wc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:17:19.206530 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.206192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b608f9-631a-4902-a9e6-81a57f218373-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:17:19.206530 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.206442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:17:19.206659 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.206536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-istio-token" (OuterVolumeSpecName: "istio-token") pod "46b608f9-631a-4902-a9e6-81a57f218373" (UID: "46b608f9-631a-4902-a9e6-81a57f218373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:17:19.305003 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.304963    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b608f9-631a-4902-a9e6-81a57f218373-istio-podinfo\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.305003 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.304996    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-workload-socket\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.305003 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.305006    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-istio-token\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.305003 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.305013    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b608f9-631a-4902-a9e6-81a57f218373-istiod-ca-cert\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.305283 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.305024    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5wc4\" (UniqueName: \"kubernetes.io/projected/46b608f9-631a-4902-a9e6-81a57f218373-kube-api-access-r5wc4\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.305283 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.305037    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b608f9-631a-4902-a9e6-81a57f218373-istio-envoy\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:19.491062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.490972    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b608f9-631a-4902-a9e6-81a57f218373" containerID="c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab" exitCode=0
Sep 16 01:17:19.491062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.491037    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"
Sep 16 01:17:19.491062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.491046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" event={"ID":"46b608f9-631a-4902-a9e6-81a57f218373","Type":"ContainerDied","Data":"c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab"}
Sep 16 01:17:19.491314 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.491111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn" event={"ID":"46b608f9-631a-4902-a9e6-81a57f218373","Type":"ContainerDied","Data":"d1210f809570dffce1f3e342a94551a61621948ef8d4eba830b467bf4c4191ad"}
Sep 16 01:17:19.491314 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.491135    2581 scope.go:117] "RemoveContainer" containerID="c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab"
Sep 16 01:17:19.501096 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.501076    2581 scope.go:117] "RemoveContainer" containerID="f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378"
Sep 16 01:17:19.510408 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.510375    2581 scope.go:117] "RemoveContainer" containerID="c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab"
Sep 16 01:17:19.510734 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:17:19.510713    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab\": container with ID starting with c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab not found: ID does not exist" containerID="c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab"
Sep 16 01:17:19.510805 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.510744    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab"} err="failed to get container status \"c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab\": rpc error: code = NotFound desc = could not find container \"c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab\": container with ID starting with c15c6b1ac99d4fc627ec1d3d4f59d40e5c124dca749872ec2269dc23e3251cab not found: ID does not exist"
Sep 16 01:17:19.510805 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.510764    2581 scope.go:117] "RemoveContainer" containerID="f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378"
Sep 16 01:17:19.511058 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:17:19.511036    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378\": container with ID starting with f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378 not found: ID does not exist" containerID="f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378"
Sep 16 01:17:19.511113 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.511068    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378"} err="failed to get container status \"f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378\": rpc error: code = NotFound desc = could not find container \"f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378\": container with ID starting with f0fcda424f1353620e93e54e703523ed70eaed0115dc3adb139dc1d0431da378 not found: ID does not exist"
Sep 16 01:17:19.512237 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.512217    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"]
Sep 16 01:17:19.515666 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.515645    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f46871-openshift-default-6d978f6dc6-km5hn"]
Sep 16 01:17:19.579941 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.579900    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:19.941565 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:19.941530    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b608f9-631a-4902-a9e6-81a57f218373" path="/var/lib/kubelet/pods/46b608f9-631a-4902-a9e6-81a57f218373/volumes"
Sep 16 01:17:20.335773 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:20.335744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:21.118370 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:21.118339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:30.780770 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:30.780684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:17:31.883221 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:31.883179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:31.983745 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:31.983706    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:33.183592 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.183550    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"]
Sep 16 01:17:33.184116 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.183776    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v" podUID="53031bfb-5aa6-41b1-a856-21781e87c274" containerName="maas-api" containerID="cri-o://9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96" gracePeriod=30
Sep 16 01:17:33.206888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.206842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:33.274456 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.274372    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:33.378212 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.378160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:33.438561 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.438531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:33.453904 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.453881    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:17:33.537987 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.537952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-metrics-certs\") pod \"53031bfb-5aa6-41b1-a856-21781e87c274\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") "
Sep 16 01:17:33.538184 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.538020    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfdhn\" (UniqueName: \"kubernetes.io/projected/53031bfb-5aa6-41b1-a856-21781e87c274-kube-api-access-nfdhn\") pod \"53031bfb-5aa6-41b1-a856-21781e87c274\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") "
Sep 16 01:17:33.538184 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.538097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls\") pod \"53031bfb-5aa6-41b1-a856-21781e87c274\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") "
Sep 16 01:17:33.538184 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.538135    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53031bfb-5aa6-41b1-a856-21781e87c274-openshift-service-ca\") pod \"53031bfb-5aa6-41b1-a856-21781e87c274\" (UID: \"53031bfb-5aa6-41b1-a856-21781e87c274\") "
Sep 16 01:17:33.538588 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.538560    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53031bfb-5aa6-41b1-a856-21781e87c274-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53031bfb-5aa6-41b1-a856-21781e87c274" (UID: "53031bfb-5aa6-41b1-a856-21781e87c274"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:17:33.540237 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.540211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53031bfb-5aa6-41b1-a856-21781e87c274" (UID: "53031bfb-5aa6-41b1-a856-21781e87c274"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:17:33.540237 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.540231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53031bfb-5aa6-41b1-a856-21781e87c274" (UID: "53031bfb-5aa6-41b1-a856-21781e87c274"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:17:33.540370 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.540218    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53031bfb-5aa6-41b1-a856-21781e87c274-kube-api-access-nfdhn" (OuterVolumeSpecName: "kube-api-access-nfdhn") pod "53031bfb-5aa6-41b1-a856-21781e87c274" (UID: "53031bfb-5aa6-41b1-a856-21781e87c274"). InnerVolumeSpecName "kube-api-access-nfdhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:17:33.568612 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.568571    2581 generic.go:356] "Generic (PLEG): container finished" podID="53031bfb-5aa6-41b1-a856-21781e87c274" containerID="9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96" exitCode=0
Sep 16 01:17:33.568739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.568630    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"
Sep 16 01:17:33.568739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.568658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v" event={"ID":"53031bfb-5aa6-41b1-a856-21781e87c274","Type":"ContainerDied","Data":"9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96"}
Sep 16 01:17:33.568739 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.568719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v" event={"ID":"53031bfb-5aa6-41b1-a856-21781e87c274","Type":"ContainerDied","Data":"44cf716064336c5afa7b00aaa1900260f53506722e302ee3004708de478555e3"}
Sep 16 01:17:33.568853 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.568744    2581 scope.go:117] "RemoveContainer" containerID="9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96"
Sep 16 01:17:33.580624 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.580452    2581 scope.go:117] "RemoveContainer" containerID="9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96"
Sep 16 01:17:33.580989 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:17:33.580955    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96\": container with ID starting with 9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96 not found: ID does not exist" containerID="9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96"
Sep 16 01:17:33.581103 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.580997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96"} err="failed to get container status \"9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96\": rpc error: code = NotFound desc = could not find container \"9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96\": container with ID starting with 9e38b674ce9d404fc99d8ac11f73e390796fdf612f2c1419bb2e7c3efd8feb96 not found: ID does not exist"
Sep 16 01:17:33.593388 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.593347    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"]
Sep 16 01:17:33.596579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.596552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-089563-56b68599bb-4sf6v"]
Sep 16 01:17:33.639566 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.639541    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.639566 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.639565    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53031bfb-5aa6-41b1-a856-21781e87c274-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.639742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.639576    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53031bfb-5aa6-41b1-a856-21781e87c274-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.639742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.639585    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdhn\" (UniqueName: \"kubernetes.io/projected/53031bfb-5aa6-41b1-a856-21781e87c274-kube-api-access-nfdhn\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.786939 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.786901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:33.854086 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.854038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:33.941109 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:33.941077    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53031bfb-5aa6-41b1-a856-21781e87c274" path="/var/lib/kubelet/pods/53031bfb-5aa6-41b1-a856-21781e87c274/volumes"
Sep 16 01:17:34.457462 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:34.457428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:35.317121 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:35.317082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:38.831853 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:38.831807    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:39.989786 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:39.989750    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:17:48.250974 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:48.250936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:17:48.304953 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:17:48.304915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:02.045594 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:02.045565    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:18:02.054519 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:02.054474    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:18:15.015982 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:15.015939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:15.679341 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:15.679307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:21.061897 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:21.061860    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:21.115305 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:21.115269    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:21.386135 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:21.386053    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:22.014764 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:22.014728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:22.920788 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:22.920749    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:23.575644 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:23.575611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:24.311413 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:24.311379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:32.167956 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.167923    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"]
Sep 16 01:18:32.168415 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.168136    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv" podUID="ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" containerName="maas-api" containerID="cri-o://209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9" gracePeriod=30
Sep 16 01:18:32.193798 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.193758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:32.255727 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.255694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:32.310378 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.310326    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:32.386311 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.386270    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:32.418412 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.418353    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:18:32.445249 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.445208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:32.482104 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.482071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnl77\" (UniqueName: \"kubernetes.io/projected/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-kube-api-access-wnl77\") pod \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") "
Sep 16 01:18:32.482300 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.482131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-openshift-service-ca\") pod \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") "
Sep 16 01:18:32.482300 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.482167    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-metrics-certs\") pod \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") "
Sep 16 01:18:32.482300 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.482214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-maas-api-tls\") pod \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\" (UID: \"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5\") "
Sep 16 01:18:32.482619 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.482582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" (UID: "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:32.484599 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.484571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-kube-api-access-wnl77" (OuterVolumeSpecName: "kube-api-access-wnl77") pod "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" (UID: "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5"). InnerVolumeSpecName "kube-api-access-wnl77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:18:32.484860 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.484827    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" (UID: "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:18:32.484956 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.484889    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" (UID: "ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:18:32.583925 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.583875    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnl77\" (UniqueName: \"kubernetes.io/projected/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-kube-api-access-wnl77\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:32.583925 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.583921    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:32.583925 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.583932    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:32.583925 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.583941    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:32.784943 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.784908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:32.887558 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.887522    2581 generic.go:356] "Generic (PLEG): container finished" podID="ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" containerID="209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9" exitCode=0
Sep 16 01:18:32.887737 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.887573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"
Sep 16 01:18:32.887737 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.887566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv" event={"ID":"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5","Type":"ContainerDied","Data":"209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9"}
Sep 16 01:18:32.887737 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.887713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv" event={"ID":"ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5","Type":"ContainerDied","Data":"d4e35c2696e803041bae2ac4218a3db16a21ea5f5154dace4ecb2674c23829fc"}
Sep 16 01:18:32.887843 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.887743    2581 scope.go:117] "RemoveContainer" containerID="209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9"
Sep 16 01:18:32.898429 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.898404    2581 scope.go:117] "RemoveContainer" containerID="209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9"
Sep 16 01:18:32.898876 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:18:32.898796    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9\": container with ID starting with 209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9 not found: ID does not exist" containerID="209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9"
Sep 16 01:18:32.898876 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.898837    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9"} err="failed to get container status \"209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9\": rpc error: code = NotFound desc = could not find container \"209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9\": container with ID starting with 209d0d317e656241faec65fde40c684100b910c824e39efa837494498ab65eb9 not found: ID does not exist"
Sep 16 01:18:32.908329 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.908302    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"]
Sep 16 01:18:32.911869 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.911844    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-195f51-7c88449b89-dwxsv"]
Sep 16 01:18:32.935282 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:32.935237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:33.608852 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:33.608775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:33.942422 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:33.942340    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" path="/var/lib/kubelet/pods/ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5/volumes"
Sep 16 01:18:36.185284 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:36.185236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:18:38.515357 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:38.515307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:38.615731 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:38.615695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:38.784747 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:38.784655    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:39.289682 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:39.289644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:40.349358 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:40.349321    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:40.837107 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:40.837068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:49.558644 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.558609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"]
Sep 16 01:18:49.559110 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.558886    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" podUID="4a0f295f-fbea-4d21-994b-746140b11108" containerName="maas-api" containerID="cri-o://a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e" gracePeriod=30
Sep 16 01:18:49.584326 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.584294    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" podUID="4a0f295f-fbea-4d21-994b-746140b11108" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.53:8443/health\": dial tcp 10.132.0.53:8443: connect: connection refused"
Sep 16 01:18:49.593311 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.593273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:49.652073 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.652039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:49.753399 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.753358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:49.806724 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.806694    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:18:49.814135 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.814049    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:49.943320 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.943288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f295f-fbea-4d21-994b-746140b11108-openshift-service-ca\") pod \"4a0f295f-fbea-4d21-994b-746140b11108\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") "
Sep 16 01:18:49.943527 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.943334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls\") pod \"4a0f295f-fbea-4d21-994b-746140b11108\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") "
Sep 16 01:18:49.943527 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.943366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-metrics-certs\") pod \"4a0f295f-fbea-4d21-994b-746140b11108\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") "
Sep 16 01:18:49.943527 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.943476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snrjf\" (UniqueName: \"kubernetes.io/projected/4a0f295f-fbea-4d21-994b-746140b11108-kube-api-access-snrjf\") pod \"4a0f295f-fbea-4d21-994b-746140b11108\" (UID: \"4a0f295f-fbea-4d21-994b-746140b11108\") "
Sep 16 01:18:49.943733 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.943702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0f295f-fbea-4d21-994b-746140b11108-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a0f295f-fbea-4d21-994b-746140b11108" (UID: "4a0f295f-fbea-4d21-994b-746140b11108"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:49.945613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.945580    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a0f295f-fbea-4d21-994b-746140b11108" (UID: "4a0f295f-fbea-4d21-994b-746140b11108"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:18:49.945613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.945596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a0f295f-fbea-4d21-994b-746140b11108" (UID: "4a0f295f-fbea-4d21-994b-746140b11108"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:18:49.945757 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.945596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0f295f-fbea-4d21-994b-746140b11108-kube-api-access-snrjf" (OuterVolumeSpecName: "kube-api-access-snrjf") pod "4a0f295f-fbea-4d21-994b-746140b11108" (UID: "4a0f295f-fbea-4d21-994b-746140b11108"). InnerVolumeSpecName "kube-api-access-snrjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:18:49.982559 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.982521    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a0f295f-fbea-4d21-994b-746140b11108" containerID="a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e" exitCode=0
Sep 16 01:18:49.982726 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.982575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" event={"ID":"4a0f295f-fbea-4d21-994b-746140b11108","Type":"ContainerDied","Data":"a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e"}
Sep 16 01:18:49.982726 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.982637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr" event={"ID":"4a0f295f-fbea-4d21-994b-746140b11108","Type":"ContainerDied","Data":"75986e05978478f68cb2088854175cf9fdf8f95abc128b63225e747cdc67e390"}
Sep 16 01:18:49.982726 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.982576    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"
Sep 16 01:18:49.982726 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.982659    2581 scope.go:117] "RemoveContainer" containerID="a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e"
Sep 16 01:18:49.992214 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.992194    2581 scope.go:117] "RemoveContainer" containerID="a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e"
Sep 16 01:18:49.992534 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:18:49.992512    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e\": container with ID starting with a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e not found: ID does not exist" containerID="a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e"
Sep 16 01:18:49.992613 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:49.992542    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e"} err="failed to get container status \"a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e\": rpc error: code = NotFound desc = could not find container \"a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e\": container with ID starting with a71c943ec933bc6d2236a851a7e49baa2b0ff54b816ad78a90ddb0ea7aad0b1e not found: ID does not exist"
Sep 16 01:18:50.003084 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.003049    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"]
Sep 16 01:18:50.006532 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.006502    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f831c5-8696d6c7bf-vpmmr"]
Sep 16 01:18:50.044385 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.044339    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snrjf\" (UniqueName: \"kubernetes.io/projected/4a0f295f-fbea-4d21-994b-746140b11108-kube-api-access-snrjf\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:50.044385 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.044373    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0f295f-fbea-4d21-994b-746140b11108-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:50.044385 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.044388    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:50.044731 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.044399    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0f295f-fbea-4d21-994b-746140b11108-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:18:50.091120 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.091038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:50.204366 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:50.204332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:51.941477 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:51.941438    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a0f295f-fbea-4d21-994b-746140b11108" path="/var/lib/kubelet/pods/4a0f295f-fbea-4d21-994b-746140b11108/volumes"
Sep 16 01:18:52.486472 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:52.486435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:18:56.131740 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:56.131639    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:56.231783 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:56.231742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:18:56.421778 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:18:56.421675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:07.308262 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.308214    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"]
Sep 16 01:19:07.308747 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.308472    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn" podUID="178f7896-2e17-4e15-a032-5577d31b98ea" containerName="maas-api" containerID="cri-o://1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189" gracePeriod=30
Sep 16 01:19:07.345053 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.345016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:07.407752 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.407714    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:07.465137 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.465106    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:07.527341 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.527304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:07.553307 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.553281    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:19:07.586295 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.586203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:07.605533 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.605471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-metrics-certs\") pod \"178f7896-2e17-4e15-a032-5577d31b98ea\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") "
Sep 16 01:19:07.605701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.605562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/178f7896-2e17-4e15-a032-5577d31b98ea-openshift-service-ca\") pod \"178f7896-2e17-4e15-a032-5577d31b98ea\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") "
Sep 16 01:19:07.605701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.605633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd8vl\" (UniqueName: \"kubernetes.io/projected/178f7896-2e17-4e15-a032-5577d31b98ea-kube-api-access-vd8vl\") pod \"178f7896-2e17-4e15-a032-5577d31b98ea\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") "
Sep 16 01:19:07.605701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.605692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls\") pod \"178f7896-2e17-4e15-a032-5577d31b98ea\" (UID: \"178f7896-2e17-4e15-a032-5577d31b98ea\") "
Sep 16 01:19:07.605973 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.605945    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178f7896-2e17-4e15-a032-5577d31b98ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "178f7896-2e17-4e15-a032-5577d31b98ea" (UID: "178f7896-2e17-4e15-a032-5577d31b98ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:19:07.607849 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.607826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "178f7896-2e17-4e15-a032-5577d31b98ea" (UID: "178f7896-2e17-4e15-a032-5577d31b98ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:19:07.607939 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.607875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "178f7896-2e17-4e15-a032-5577d31b98ea" (UID: "178f7896-2e17-4e15-a032-5577d31b98ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:19:07.607939 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.607882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178f7896-2e17-4e15-a032-5577d31b98ea-kube-api-access-vd8vl" (OuterVolumeSpecName: "kube-api-access-vd8vl") pod "178f7896-2e17-4e15-a032-5577d31b98ea" (UID: "178f7896-2e17-4e15-a032-5577d31b98ea"). InnerVolumeSpecName "kube-api-access-vd8vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:19:07.706538 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.706471    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-maas-api-tls\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:19:07.706538 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.706528    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/178f7896-2e17-4e15-a032-5577d31b98ea-metrics-certs\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:19:07.706538 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.706537    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/178f7896-2e17-4e15-a032-5577d31b98ea-openshift-service-ca\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:19:07.706538 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.706547    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8vl\" (UniqueName: \"kubernetes.io/projected/178f7896-2e17-4e15-a032-5577d31b98ea-kube-api-access-vd8vl\") on node \"ip-10-0-130-4.ec2.internal\" DevicePath \"\""
Sep 16 01:19:07.923207 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:07.923114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:08.081035 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.080990    2581 generic.go:356] "Generic (PLEG): container finished" podID="178f7896-2e17-4e15-a032-5577d31b98ea" containerID="1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189" exitCode=0
Sep 16 01:19:08.081205 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.081046    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"
Sep 16 01:19:08.081205 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.081065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn" event={"ID":"178f7896-2e17-4e15-a032-5577d31b98ea","Type":"ContainerDied","Data":"1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189"}
Sep 16 01:19:08.081205 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.081117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn" event={"ID":"178f7896-2e17-4e15-a032-5577d31b98ea","Type":"ContainerDied","Data":"a0fbde4d04c2d9d32b2aadcfe3ce44104c140144b809e5bd3232b389996571d4"}
Sep 16 01:19:08.081205 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.081142    2581 scope.go:117] "RemoveContainer" containerID="1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189"
Sep 16 01:19:08.090514 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.090469    2581 scope.go:117] "RemoveContainer" containerID="1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189"
Sep 16 01:19:08.090809 ip-10-0-130-4 kubenswrapper[2581]: E0916 01:19:08.090789    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189\": container with ID starting with 1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189 not found: ID does not exist" containerID="1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189"
Sep 16 01:19:08.090865 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.090817    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189"} err="failed to get container status \"1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189\": rpc error: code = NotFound desc = could not find container \"1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189\": container with ID starting with 1a58135d1e2d0e8791a82fe3fdfc5859c5f07659ce2384cd5ef7df372c76c189 not found: ID does not exist"
Sep 16 01:19:08.096114 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.096089    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"]
Sep 16 01:19:08.102011 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:08.101987    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ee1bf-757b8bc65f-hwcwn"]
Sep 16 01:19:09.878214 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:09.878165    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:19:09.941631 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:09.941589    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="178f7896-2e17-4e15-a032-5577d31b98ea" path="/var/lib/kubelet/pods/178f7896-2e17-4e15-a032-5577d31b98ea/volumes"
Sep 16 01:19:13.833185 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:13.833144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:13.934583 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:13.934541    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:14.110692 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:14.110601    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:18.080526 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:18.080470    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:19:33.457075 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:33.457039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:19:33.511013 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:19:33.510967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:20:18.602701 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:20:18.602665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:20:39.960596 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:20:39.960517    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:20:40.983848 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:20:40.983811    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:20:41.529530 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:20:41.529465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:21:06.283060 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:06.283018    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:21:15.888248 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:15.888195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:21:23.283596 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:23.283562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:21:30.979549 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:30.979512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:21:41.172816 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:41.172774    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:21:41.273527 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:41.273467    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:21:46.086341 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:46.086305    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:21:56.183758 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:21:56.183667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:22:00.565794 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:22:00.565757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:22:02.275041 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:22:02.275008    2581 scope.go:117] "RemoveContainer" containerID="a89c7ea99e34aea3bf9bb492c2923c73276eec9466998f56a1a1d19d5c905b5b"
Sep 16 01:23:02.078976 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:02.078944    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:23:02.088792 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:02.088761    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:23:10.743211 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:10.743169    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:23:10.843886 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:10.843849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:23:24.679751 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:24.679654    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:23:32.888053 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:32.888014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:23:39.382780 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:39.382741    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:23:47.008202 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:47.008162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:23:56.478861 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:56.478827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:23:59.387186 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:23:59.387145    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:24:19.403759 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:24:19.403721    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:24:22.282328 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:24:22.282285    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:24:40.718052 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:24:40.718010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:24:42.589565 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:24:42.589522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:24:43.181271 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:24:43.181231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:24:52.976978 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:24:52.976939    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:25:00.886558 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:25:00.886443    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:25:03.555941 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:25:03.555905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:28:02.112235 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:02.112145    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:28:02.124092 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:02.124068    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:28:06.879819 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:06.879784    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:28:14.076764 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:14.076718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:28:20.082053 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:20.082011    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:28:27.490473 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:27.490429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-knfs4"]
Sep 16 01:28:30.089050 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089014    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xll2x/must-gather-7cltv"]
Sep 16 01:28:30.089550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089496    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53031bfb-5aa6-41b1-a856-21781e87c274" containerName="maas-api"
Sep 16 01:28:30.089550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089514    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="53031bfb-5aa6-41b1-a856-21781e87c274" containerName="maas-api"
Sep 16 01:28:30.089550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089528    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy"
Sep 16 01:28:30.089550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089534    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy"
Sep 16 01:28:30.089550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089545    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" containerName="maas-api"
Sep 16 01:28:30.089550 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089551    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089568    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089574    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089579    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089589    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178f7896-2e17-4e15-a032-5577d31b98ea" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089594    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="178f7896-2e17-4e15-a032-5577d31b98ea" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089602    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a0f295f-fbea-4d21-994b-746140b11108" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089607    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a0f295f-fbea-4d21-994b-746140b11108" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089671    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089680    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="53031bfb-5aa6-41b1-a856-21781e87c274" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089686    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="178f7896-2e17-4e15-a032-5577d31b98ea" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089693    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebbc7248-c53e-4f76-a6a9-3e7b13fb4df5" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089699    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089705    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a0f295f-fbea-4d21-994b-746140b11108" containerName="maas-api"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089710    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b608f9-631a-4902-a9e6-81a57f218373" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089774    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089779    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy"
Sep 16 01:28:30.089884 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.089842    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="484f419f-4ff6-4511-801f-e50af17337b3" containerName="istio-proxy"
Sep 16 01:28:30.092973 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.092956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.095230 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.095202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xll2x\"/\"openshift-service-ca.crt\""
Sep 16 01:28:30.095230 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.095226    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xll2x\"/\"kube-root-ca.crt\""
Sep 16 01:28:30.096021 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.096004    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xll2x\"/\"default-dockercfg-fdxcj\""
Sep 16 01:28:30.108990 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.108961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xll2x/must-gather-7cltv"]
Sep 16 01:28:30.141395 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.141363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:28:30.160631 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.160604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/95b9e737-c6ff-474e-a902-0f76493ed706-kube-api-access-hnpxg\") pod \"must-gather-7cltv\" (UID: \"95b9e737-c6ff-474e-a902-0f76493ed706\") " pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.160777 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.160679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95b9e737-c6ff-474e-a902-0f76493ed706-must-gather-output\") pod \"must-gather-7cltv\" (UID: \"95b9e737-c6ff-474e-a902-0f76493ed706\") " pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.197000 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.196961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log" limit=1024
Sep 16 01:28:30.261542 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.261503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95b9e737-c6ff-474e-a902-0f76493ed706-must-gather-output\") pod \"must-gather-7cltv\" (UID: \"95b9e737-c6ff-474e-a902-0f76493ed706\") " pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.261726 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.261555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/95b9e737-c6ff-474e-a902-0f76493ed706-kube-api-access-hnpxg\") pod \"must-gather-7cltv\" (UID: \"95b9e737-c6ff-474e-a902-0f76493ed706\") " pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.261861 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.261839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95b9e737-c6ff-474e-a902-0f76493ed706-must-gather-output\") pod \"must-gather-7cltv\" (UID: \"95b9e737-c6ff-474e-a902-0f76493ed706\") " pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.268952 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.268912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/95b9e737-c6ff-474e-a902-0f76493ed706-kube-api-access-hnpxg\") pod \"must-gather-7cltv\" (UID: \"95b9e737-c6ff-474e-a902-0f76493ed706\") " pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.402697 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.402601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xll2x/must-gather-7cltv"
Sep 16 01:28:30.671774 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.671709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xll2x/must-gather-7cltv"]
Sep 16 01:28:30.674168 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:28:30.674137    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95b9e737_c6ff_474e_a902_0f76493ed706.slice/crio-be6bca01066fef8d3b311fe7e3d0136e9cae43f4c9780d37874d03a9bd0366fd WatchSource:0}: Error finding container be6bca01066fef8d3b311fe7e3d0136e9cae43f4c9780d37874d03a9bd0366fd: Status 404 returned error can't find the container with id be6bca01066fef8d3b311fe7e3d0136e9cae43f4c9780d37874d03a9bd0366fd
Sep 16 01:28:30.676491 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.676453    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 01:28:30.932680 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:30.932592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xll2x/must-gather-7cltv" event={"ID":"95b9e737-c6ff-474e-a902-0f76493ed706","Type":"ContainerStarted","Data":"be6bca01066fef8d3b311fe7e3d0136e9cae43f4c9780d37874d03a9bd0366fd"}
Sep 16 01:28:31.945742 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:31.945697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xll2x/must-gather-7cltv" event={"ID":"95b9e737-c6ff-474e-a902-0f76493ed706","Type":"ContainerStarted","Data":"e37972e96301726ce8b754ebab6fe5ce1ee9baccfad044c5370cc79ea922154e"}
Sep 16 01:28:31.946140 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:31.945756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xll2x/must-gather-7cltv" event={"ID":"95b9e737-c6ff-474e-a902-0f76493ed706","Type":"ContainerStarted","Data":"f0c65b484ada1d762cbe8ae3d81140ab38aedf6cb4ab8925fa4bcca35742a59e"}
Sep 16 01:28:33.022268 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:33.022234    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pxhn7_dad4e663-1c7e-4bfc-8dc4-f7ec0c5d6ceb/global-pull-secret-syncer/0.log"
Sep 16 01:28:33.169549 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:33.169517    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kwslh_d48fcf55-35a0-4c4b-a7c8-2a5ecaefc7e7/konnectivity-agent/0.log"
Sep 16 01:28:33.280340 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:33.280264    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-4.ec2.internal_3855482ecf545e291be557b4f332fc2e/haproxy/0.log"
Sep 16 01:28:38.060846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:38.059663    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84cf54b799-x42l7_1e0c0b69-1026-44a9-b250-c54c6ed9799c/authorino/0.log"
Sep 16 01:28:38.164329 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:38.164288    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pz6vc_8532120d-2bb1-49d3-a693-c4a3190d4e39/kuadrant-console-plugin/0.log"
Sep 16 01:28:38.232676 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:38.232635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-knfs4_5008c036-32a3-413a-b759-6a7f3df357f0/limitador/0.log"
Sep 16 01:28:39.819226 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:39.819191    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7229b5f4-c8ec-409b-9a7a-3b34b04c1a15/alertmanager/0.log"
Sep 16 01:28:39.845878 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:39.845843    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7229b5f4-c8ec-409b-9a7a-3b34b04c1a15/config-reloader/0.log"
Sep 16 01:28:39.867505 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:39.867456    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7229b5f4-c8ec-409b-9a7a-3b34b04c1a15/kube-rbac-proxy-web/0.log"
Sep 16 01:28:39.888908 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:39.888875    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7229b5f4-c8ec-409b-9a7a-3b34b04c1a15/kube-rbac-proxy/0.log"
Sep 16 01:28:39.913050 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:39.913018    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7229b5f4-c8ec-409b-9a7a-3b34b04c1a15/kube-rbac-proxy-metric/0.log"
Sep 16 01:28:39.933812 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:39.933778    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7229b5f4-c8ec-409b-9a7a-3b34b04c1a15/prom-label-proxy/0.log"
Sep 16 01:28:39.957140 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:39.957112    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7229b5f4-c8ec-409b-9a7a-3b34b04c1a15/init-config-reloader/0.log"
Sep 16 01:28:40.039216 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.039182    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-t87km_3f93155b-bfb8-4840-a141-bf9418701d29/kube-state-metrics/0.log"
Sep 16 01:28:40.059373 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.059341    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-t87km_3f93155b-bfb8-4840-a141-bf9418701d29/kube-rbac-proxy-main/0.log"
Sep 16 01:28:40.083579 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.083461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-t87km_3f93155b-bfb8-4840-a141-bf9418701d29/kube-rbac-proxy-self/0.log"
Sep 16 01:28:40.141320 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.141288    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-v9t45_3c8d4a23-f845-47d0-b31f-6045a3154f7f/monitoring-plugin/0.log"
Sep 16 01:28:40.172506 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.172039    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9c6n2_f16b75b6-03e7-4fdd-9f23-359f953e275b/node-exporter/0.log"
Sep 16 01:28:40.196505 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.195497    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9c6n2_f16b75b6-03e7-4fdd-9f23-359f953e275b/kube-rbac-proxy/0.log"
Sep 16 01:28:40.224354 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.224320    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9c6n2_f16b75b6-03e7-4fdd-9f23-359f953e275b/init-textfile/0.log"
Sep 16 01:28:40.444891 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.444857    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pzmhb_ed050320-b082-466b-b27a-0ffc70a5f972/kube-rbac-proxy-main/0.log"
Sep 16 01:28:40.469703 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.469671    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pzmhb_ed050320-b082-466b-b27a-0ffc70a5f972/kube-rbac-proxy-self/0.log"
Sep 16 01:28:40.496254 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.496032    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pzmhb_ed050320-b082-466b-b27a-0ffc70a5f972/openshift-state-metrics/0.log"
Sep 16 01:28:40.698067 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.697971    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rnd5b_1a3b3e9c-e324-48e2-8302-6ff24fbf9463/prometheus-operator/0.log"
Sep 16 01:28:40.716927 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.716837    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rnd5b_1a3b3e9c-e324-48e2-8302-6ff24fbf9463/kube-rbac-proxy/0.log"
Sep 16 01:28:40.772915 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.772869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587678f467-2xt4g_2b9bfcc2-9fb6-4719-af15-270818df9acf/telemeter-client/0.log"
Sep 16 01:28:40.792324 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.792294    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587678f467-2xt4g_2b9bfcc2-9fb6-4719-af15-270818df9acf/reload/0.log"
Sep 16 01:28:40.812399 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:40.812365    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587678f467-2xt4g_2b9bfcc2-9fb6-4719-af15-270818df9acf/kube-rbac-proxy/0.log"
Sep 16 01:28:42.414001 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:42.413966    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-j45sf_cd6a62e3-f986-4971-8642-42861233c1b6/networking-console-plugin/0.log"
Sep 16 01:28:42.983600 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:42.983557    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/2.log"
Sep 16 01:28:42.990752 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:42.990710    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-c4jbm_0187d13e-b0d9-40f1-91d1-0aea6ba2be15/console-operator/3.log"
Sep 16 01:28:43.216438 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.216364    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xll2x/must-gather-7cltv" podStartSLOduration=12.380879254 podStartE2EDuration="13.216336921s" totalImagesPullingTime="835.457667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:28:31.960331038 +0000 UTC m=+2730.503468012" watchObservedRunningTime="2026-09-16 01:28:43.216336921 +0000 UTC m=+2741.759473897"
Sep 16 01:28:43.218118 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.218083    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"]
Sep 16 01:28:43.222741 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.222713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.232054 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.231571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"]
Sep 16 01:28:43.303573 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.303519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-proc\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.303573 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.303575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-lib-modules\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.303839 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.303652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-podres\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.303839 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.303804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-sys\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.303963 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.303851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7hd\" (UniqueName: \"kubernetes.io/projected/34d932dd-446d-4473-8ecd-f58905000117-kube-api-access-zg7hd\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.404625 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.404582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-sys\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.404625 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.404631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7hd\" (UniqueName: \"kubernetes.io/projected/34d932dd-446d-4473-8ecd-f58905000117-kube-api-access-zg7hd\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.404908 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.404685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-proc\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.404908 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.404709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-lib-modules\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.404908 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.404753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-podres\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.405090 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.404972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-podres\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.405090 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.405022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-sys\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.405090 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.405066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-lib-modules\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.405257 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.405136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34d932dd-446d-4473-8ecd-f58905000117-proc\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.412846 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.412804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7hd\" (UniqueName: \"kubernetes.io/projected/34d932dd-446d-4473-8ecd-f58905000117-kube-api-access-zg7hd\") pod \"perf-node-gather-daemonset-cp6vf\" (UID: \"34d932dd-446d-4473-8ecd-f58905000117\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.537774 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.537738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:43.606548 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.606395    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-889kn_7e70c35a-bd71-4fdf-adb8-8541124d237b/download-server/0.log"
Sep 16 01:28:43.824858 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:43.824651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"]
Sep 16 01:28:43.827475 ip-10-0-130-4 kubenswrapper[2581]: W0916 01:28:43.827437    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod34d932dd_446d_4473_8ecd_f58905000117.slice/crio-d444358f436e7922663ffcf470b0cf61c37c873311777c5c7d70ac5a0a379324 WatchSource:0}: Error finding container d444358f436e7922663ffcf470b0cf61c37c873311777c5c7d70ac5a0a379324: Status 404 returned error can't find the container with id d444358f436e7922663ffcf470b0cf61c37c873311777c5c7d70ac5a0a379324
Sep 16 01:28:44.039497 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:44.039453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf" event={"ID":"34d932dd-446d-4473-8ecd-f58905000117","Type":"ContainerStarted","Data":"763c351380340b7f7fabeb5f2194f4b758095b96f9ab0896bbe1e68a5f54e9a9"}
Sep 16 01:28:44.039699 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:44.039534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf" event={"ID":"34d932dd-446d-4473-8ecd-f58905000117","Type":"ContainerStarted","Data":"d444358f436e7922663ffcf470b0cf61c37c873311777c5c7d70ac5a0a379324"}
Sep 16 01:28:44.039699 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:44.039643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:44.056349 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:44.056286    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf" podStartSLOduration=1.056263391 podStartE2EDuration="1.056263391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:28:44.05352227 +0000 UTC m=+2742.596659243" watchObservedRunningTime="2026-09-16 01:28:44.056263391 +0000 UTC m=+2742.599400367"
Sep 16 01:28:45.504786 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:45.504753    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hs5c9_2466104a-6c7b-4788-b0fb-0a7d6144e3d4/dns/0.log"
Sep 16 01:28:45.525342 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:45.525300    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hs5c9_2466104a-6c7b-4788-b0fb-0a7d6144e3d4/kube-rbac-proxy/0.log"
Sep 16 01:28:45.591213 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:45.591172    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-46zrq_6ebf9614-24d3-4cc7-8fb9-a1d6c3d92405/dns-node-resolver/0.log"
Sep 16 01:28:46.158203 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:46.158164    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8z8zz_d8b1989d-a634-4622-82c7-9ba42c0b829b/node-ca/0.log"
Sep 16 01:28:47.540005 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:47.539972    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfdbc6f8d-xj5xh_c6fb763b-3b45-44b3-a733-3ac5f23ed8c9/kube-auth-proxy/0.log"
Sep 16 01:28:48.478856 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:48.478813    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cj78c_45b30463-b1d8-4f57-8f94-80411105dd84/serve-healthcheck-canary/0.log"
Sep 16 01:28:48.958100 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:48.958058    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-q2972_a63383b4-49c8-447e-982f-7cbf0fa0df04/insights-operator/1.log"
Sep 16 01:28:48.958585 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:48.958280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-q2972_a63383b4-49c8-447e-982f-7cbf0fa0df04/insights-operator/0.log"
Sep 16 01:28:49.043402 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:49.043374    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fdzml_faa5c9f6-fd29-4380-a883-42cbb3b78c2f/kube-rbac-proxy/0.log"
Sep 16 01:28:49.063528 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:49.063495    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fdzml_faa5c9f6-fd29-4380-a883-42cbb3b78c2f/exporter/0.log"
Sep 16 01:28:49.086123 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:49.086090    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fdzml_faa5c9f6-fd29-4380-a883-42cbb3b78c2f/extractor/0.log"
Sep 16 01:28:50.056284 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:50.056255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-cp6vf"
Sep 16 01:28:52.000029 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:51.999975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778fc5bd-x56vf_2a068eb0-afe6-4138-8813-c8f2164881c4/manager/0.log"
Sep 16 01:28:52.019847 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:52.019819    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778fc5bd-x56vf_2a068eb0-afe6-4138-8813-c8f2164881c4/copy-manifests/0.log"
Sep 16 01:28:53.587053 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:53.587022    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54b79494c6-wbhdz_a83eef67-8cdc-4413-9cd3-3732a1f00ba4/manager/0.log"
Sep 16 01:28:54.419264 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:54.419210    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfdbc6f8d-xj5xh_c6fb763b-3b45-44b3-a733-3ac5f23ed8c9/kube-auth-proxy/0.log"
Sep 16 01:28:56.300888 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:56.300833    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfdbc6f8d-xj5xh_c6fb763b-3b45-44b3-a733-3ac5f23ed8c9/kube-auth-proxy/0.log"
Sep 16 01:28:58.544195 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:58.544160    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pmsdb_11cdd715-6299-4fa9-942d-064115a5829f/migrator/0.log"
Sep 16 01:28:58.568018 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:28:58.567987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pmsdb_11cdd715-6299-4fa9-942d-064115a5829f/graceful-termination/0.log"
Sep 16 01:29:00.208407 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.208375    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x62zx_50933fa6-d2ac-4292-ba25-af08d8d200ba/kube-multus-additional-cni-plugins/0.log"
Sep 16 01:29:00.230222 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.230192    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x62zx_50933fa6-d2ac-4292-ba25-af08d8d200ba/egress-router-binary-copy/0.log"
Sep 16 01:29:00.248624 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.248592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x62zx_50933fa6-d2ac-4292-ba25-af08d8d200ba/cni-plugins/0.log"
Sep 16 01:29:00.268062 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.268036    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x62zx_50933fa6-d2ac-4292-ba25-af08d8d200ba/bond-cni-plugin/0.log"
Sep 16 01:29:00.287278 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.287249    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x62zx_50933fa6-d2ac-4292-ba25-af08d8d200ba/routeoverride-cni/0.log"
Sep 16 01:29:00.306703 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.306670    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x62zx_50933fa6-d2ac-4292-ba25-af08d8d200ba/whereabouts-cni-bincopy/0.log"
Sep 16 01:29:00.326078 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.326034    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x62zx_50933fa6-d2ac-4292-ba25-af08d8d200ba/whereabouts-cni/0.log"
Sep 16 01:29:00.473332 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.473240    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v9fk6_30dc45cb-9ef0-411a-88b1-2673963e5977/kube-multus/0.log"
Sep 16 01:29:00.533312 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.533280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mftnx_0d9cbade-9182-42ac-8e70-5fd8626eb8f3/network-metrics-daemon/0.log"
Sep 16 01:29:00.550310 ip-10-0-130-4 kubenswrapper[2581]: I0916 01:29:00.550282    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mftnx_0d9cbade-9182-42ac-8e70-5fd8626eb8f3/kube-rbac-proxy/0.log"